- Compass-ci - mailweb.openeuler.org

review star
by Wu Fengguang 02 Apr '21

02 Apr '21

Luan Shengde is our review star, thanks Luan Shengde! baijing@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 81 11 11 25 Cao Xueliang 59 12 19 12 Wei Jihui 24 13 28 5 Luan Shengde 22 7 6 3 Cui Yili 19 3 4 3 Wu Zhende 16 3 5 2 Yu Chuan 12 5 5 2 Xu Xijian 2 1 1 1 Wu Fengguang 25 0 0 1 Li Ping 17 1 1 0 Wang Chenglong 20 0 0 0 Liu Yinsi 6 0 0 0 Lu Weitao 1 0 0 0 Zhang Dewan 20 1 1 0 Wang Yong 6 0 0 0 Li Yuanchao 1 0 0 0 Bai Jing 9 4 4 0 Lin Jiaxin 18 5 4 0 Liu Shaofei

2 1

help star
by Wu Fengguang 02 Apr '21

02 Apr '21

Wu Zhende, Yu Chuan is our help star again, thanks Wu Zhende, Yu Chuan! baijing@crystal ~/todo/bin% ./help-star ../people/*/credit/2021-04-01.md 14.3 Wu Zhende 14.2 Yu Chuan 14.0 Wei Jihui 11.3 Wu Fengguang 11.0 Liu Shaofei 10.7 Wang Yong 10.2 Zhang Dewan 9.4 Lin Jiaxin 7.5 Luan Shengde 6.8 Cao Xueliang 5.8 Liu Yinsi 5.4 Bai Jing 4.4 Lu Weitao 4.2 Du Kaitian 4.0 Li Ping 2.2 Cui Yili 1.8 Wang Chenglong 1.5 Li Yuanchao 1.4 Xu Xijian 0.0 Zhang Yale

1 0

git count
by Wu Fengguang 02 Apr '21

02 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 19 Cao Xueliang 9 Liu Yinsi 8 Wu Zhende 6 Liu Shaofei 6 Lu Weitao 6 Wang Yong 4 Xu Xijian 4 Cui Yili 4 Luan Shengde 3 Wei Jihui 3 Yu Chuan 3 Li Yuanchao 3 Lin Jiaxin 3 Li Ping 2 Wang Chenglong 1 Bai Jing 1 Zhang Dewan 0 Du Kaitian 0 Wu Fengguang 0 Zhang Yale

1 0

[PATCH lkp-tests] test/ansible_test: modify ansible code to output the module name
by Wang Chenglong 02 Apr '21

02 Apr '21

[Why] modify ansible code to output the name of the module for tasks execution. [How] print("{'action':'%s'}" % task_result.__dict__['_task_fields']['action']) {'action':'package'} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/ansible_test b/tests/ansible_test index 56926bbed..1c19c2b0f 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -58,7 +58,18 @@ ansible_playbook_run() EOF [ $? -eq 0 ] && { echo "Playbook has been generated!" - } + } + [ $(command -v ansible-playbook) ] || { + echo "ansible-playbook: command not found" + exit 1 + } + + sed -i "536a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py + + sed -i "550a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py + ansible-playbook "$ansible_path"/site.yml if [ $? -eq 0 ]; then -- 2.23.0

2 1

[PATCH v2 lkp-tests] stats/ansible_test: optimize the output of error_id
by Wang Chenglong 02 Apr '21

02 Apr '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 101 ++++++++++++++++++++++++++++++++++++++------- 1 file changed, 85 insertions(+), 16 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..d7f084027 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,109 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + # when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Could not find .*):/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end + +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case i + when /(.*)/ + output_error $1 + end + end +end + while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + when /(FAILED!|failed:).*=>(.*)/ - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? - ansible_failed_json = JSON.parse @ansible_failed_info + ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

2 1

[PATCH compass-ci] service/scheduler: modify the data saved in the testbox
by Wu Zhende 02 Apr '21

02 Apr '21

1. add name: for search in es 2. add type: vm/dc/physical 3. delete history content: no need this info testbox's history info can find in logging Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 19 ++++++++++++++++++- src/scheduler/elasticsearch_client.cr | 7 ++----- 2 files changed, 20 insertions(+), 6 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 305d070..2e4fd30 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -160,18 +160,35 @@ class Sched state = "requesting" end + type = get_type(testbox) queues = JSON.parse(queues.to_json) hash = { "job_id" => job_id, "state" => state, "time" => get_time, "deadline" => deadline, - "queues" => queues + "queues" => queues, + "type" => type, + "name" => testbox } + @redis.update_wtmp(testbox.to_s, hash) @es.update_tbox(testbox.to_s, hash) end + def get_type(testbox) + return unless testbox + + if testbox.includes?("vm") + type = "vm" + elsif testbox.includes?("dc") + type = "dc" + else + type = "physical" + end + type + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/elasticsearch_client.cr b/src/scheduler/elasticsearch_client.cr index 219cf3d..f32256b 100644 --- a/src/scheduler/elasticsearch_client.cr +++ b/src/scheduler/elasticsearch_client.cr @@ -117,17 +117,14 @@ class Elasticsearch::Client result = @client.get_source(query) raise result unless result.is_a?(JSON::Any) - history = result["history"].as_a? result = result.as_h else result = wtmp_hash end - history ||= [] of JSON::Any - history << JSON.parse(wtmp_hash.to_json) unless wtmp_hash["state"]?.to_s == "requesting" - history = JSON.parse(history.to_json) - + history = JSON::Any.new([] of JSON::Any) body = { "history" => history} + body.any_merge!(result) body.any_merge!(wtmp_hash) -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/rpmbuild-pkg: add parameters annotation
by Wang Yong 02 Apr '21

02 Apr '21

it's for choosing parameters clearly Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 7a101b6a0..a91ae4a08 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -3,6 +3,7 @@ # - compat_os # - repo_name # - repo_addr +# Git repo jobs only use the first two parameters, SRPMs jobs use all . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -- 2.23.0

1 0

[PATCH v2 lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 02 Apr '21

02 Apr '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 101 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..706b5b249 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,101 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

3 3

[PATCH compass-ci] sched: stop ectd watcher before close etcd client
by Cao Xueliang 01 Apr '21

01 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 43 +++++++++++++++++++++++----------- 1 file changed, 29 insertions(+), 14 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 6e8d5d1..74a27af 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -129,13 +129,26 @@ class Sched jobs += job.kvs end + ec.close + return jobs, revisions.min end def consume_by_watch(queues, revision) ready_queues = split_ready_queues(queues) - channel = watch_queues(ready_queues, revision) - loop_handle_event(channel) + + channel = Channel(Array(Etcd::Model::WatchEvent)).new + ech = Hash(EtcdClient, Etcd::Watch::Watcher).new + ready_queues.each do |queue| + ec = EtcdClient.new + watcher = ec.watch_prefix(queue, start_revision: revision.to_i64, progress_notify: false, filters: [Etcd::Watch::Filter::NODELETE]) do |events| + channel.send(events) + end + ech[ec] = watcher + end + + watchers = start_watcher(ech) + loop_handle_event(channel, ech) end def split_ready_queues(queues) @@ -148,25 +161,25 @@ class Sched ready_queues.uniq end - def watch_queues(queues, revision) - puts "watch #{queues}, revision #{revision}" - channel = Channel(Array(Etcd::Model::WatchEvent)).new - queues.each do |queue| - watcher = EtcdClient.new.watch_prefix(queue, start_revision: revision.to_i64, progress_notify: false ,filters: [Etcd::Watch::Filter::NODELETE]) do |events| - channel.send(events) - end + def start_watcher(ech) + ech.each do |ec, watcher| spawn { watcher.start } Fiber.yield end - - return channel end - def loop_handle_event(channel) + def loop_handle_event(channel, ech) while true events = channel.receive events.each do |event| - return event.kv if ready2process(event.kv) + if ready2process(event.kv) + ech.each do |ec, watcher| + watcher.stop + ec.close + end + + return event.kv + end end end end @@ -176,7 +189,9 @@ class Sched f_queue = job.key t_queue = f_queue.gsub("/ready/", "/in_process/") value = job.value - ec.move(f_queue, t_queue, value) + res = ec.move(f_queue, t_queue, value) + ec.close + return res end def get_job_boot(host, boot_type) -- 2.23.0

1 0

[PATCH compass-ci] sched: close etcd client in every http request
by Cao Xueliang 01 Apr '21

01 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/etcd_client.cr | 4 ++++ src/lib/sched.cr | 4 ++++ src/scheduler/scheduler.cr | 4 ++++ 3 files changed, 12 insertions(+) diff --git a/src/lib/etcd_client.cr b/src/lib/etcd_client.cr index 7891eb3..e0d4239 100644 --- a/src/lib/etcd_client.cr +++ b/src/lib/etcd_client.cr @@ -13,6 +13,10 @@ class EtcdClient @etcd = Etcd.client(host, port, version) end + def close + @etcd.close + end + def put(queue, content) queue = "#{BASE}/#{queue}" unless queue.starts_with?(BASE) @etcd.kv.put_not_exists(queue, content) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6fd6657..7e01221 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -49,6 +49,10 @@ class Sched @log.info(%({"from": "#{(a)env.request.remote_address}", "response": #{response.to_json}})) end + def etcd_close + @etcd.close + end + def alive(version) debug_message("Env= {\n#{`export`}}") "LKP Alive! The time is #{Time.local}, version = #{version}" diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 6b032ee..ee13fd3 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -41,6 +41,10 @@ module Scheduler env.create_sched end + after_all do |env| + env.sched.etcd_close + end + # echo alive get "/" do |env| env.sched.alive(VERSION) -- 2.23.0

1 0

[PATCH compass-ci] providers/my-docker: use dc-8g for universality
by Lin Jiaxin 01 Apr '21

01 Apr '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- providers/my-docker | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/my-docker b/providers/my-docker index c8420c7..0a9f8b0 100755 --- a/providers/my-docker +++ b/providers/my-docker @@ -5,7 +5,7 @@ require_relative './docker/docker' -tbox_group = 'dc-1g' +tbox_group = 'dc-8g' hostname = "#{tbox_group}.#{ENV['USER']}-#{Process.pid}" # specify which queues will be request, use "," to separate more than 2 values queues = "#{tbox_group}~#{ENV['USER']}" -- 2.23.0

1 0

[PATCH v3 compass-ci] doc: update sparrow README.md
by Liu Yinsi 01 Apr '21

01 Apr '21

Locally deploy compass-ci just for sparrow, if users want to use other services, prompt users to apply for account and borrow machine. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 60 ++++++++++++++++++----------------------------- 1 file changed, 23 insertions(+), 37 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index d7b8e81..532b7f9 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -41,7 +41,7 @@ 3. 创建工作目录并克隆 compass-ci 项目代码 ```bash mkdir /c/ && ln -s /c/compass-ci /c/cci - git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/compass-ci.git ``` 4. 编辑setup.yaml配置用户名和邮箱 @@ -57,53 +57,39 @@ cd compass-ci/sparrow && ./install-tiny ``` -#### 提交测试任务前的准备 - -1. 测试环境是否可以提交job测试 +#### 提交测试任务 +本文以/c/lkp-tests/jobs/目录下已有的通用测试用例host-info.yaml为例 +- 使环境变量生效 ```bash - submit iperf.yaml - ``` - - 执行上述命令正常情况下会提示信息如下: + source /etc/profile.d/compass-ci ``` - submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - ``` - compass-ci搭建完毕，下面就可以开始进行测试了。 - -#### 提交测试任务到本地compass-ci -本文以测试用例iperf.yaml为例 -1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - -2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… - -3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… +- 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash - submit iperf.yaml + submit host-info.yaml ``` -#### 运行测试任务并查看任务结果 - -1. 运行测试任务 - ```bash - cd /c/compass-ci/providers/ && ./my-qemu.sh + 执行上述命令会打印提示信息如下: + ``` + submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 + submit /c/lkp-tests/jobs/host-info.yaml, got job id=nolab.1 ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +- 查看任务结果 +等待约1分钟，可根据上一步骤中打印的job id查看任务结果。 ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 - cat output + cd $(es-find id=nolab.1 |grep result_root|awk -F '"' '{print "/srv/"$4}') && ls ``` - >**说明：** - >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… + 结果文件介绍 + job.yaml文件 + job.yaml 文件中部分字段是用户提交上来的，其他字段是平台根据提交的 job 自动添加进来的。此文件包含了测试任务需要的所有参数。 -# FAQ + output文件 + output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 -* 选择下载os rootfs + stats.json + 测试用例执行完成会生成一个与测试用例同名的文件，记录它们的测试命令及标准化输出结果。compass-ci 会对这些文件进行解析，生成后缀名是 .json 的文件。 + stats.json 是所有的 json 文件的汇总，所有测试命令的关键结果都会统计到这个文件中，便于后续的比较和分析。 - 启动qemu测试机需要[os rootfs文件](http://api.compass-ci.openeuler.org:11304/os/)， - 一键部署默认下载os rootfs为openeuler aarch64 20.03，当部署完毕后，可在该目录下查看: /srv/os/openeuler/aarch64/20.03 - 如需要其他os rootfs，可使用该脚本下载: /c/compass-ci/sbin/download-rootfs +体验更多功能例如[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md)、[调测环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/log-in-machine-debug.md)、[测试结果分析](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compare-results.zh.md)等，请[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.zh.md)、[申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/borrow-machine.zh.md)。 -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: add queues to testbox
by Wu Zhende 01 Apr '21

01 Apr '21

Save data like this "queues":["sched/dc-1g~wuzhende/ready"] to ES. So we can collect testbox information of a queue. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 6 ++++-- src/scheduler/find_job_boot.cr | 2 +- 2 files changed, 5 insertions(+), 3 deletions(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 6fd6657..c84b3f3 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -143,7 +143,7 @@ class Sched @log.warn(e) end - def set_lifecycle(job, testbox) + def set_lifecycle(job, testbox, queues) if job deadline = job.get_deadline job["deadline"] = deadline @@ -156,11 +156,13 @@ class Sched state = "requesting" end + queues = JSON.parse(queues.to_json) hash = { "job_id" => job_id, "state" => state, "time" => get_time, - "deadline" => deadline + "deadline" => deadline, + "queues" => queues } @redis.update_wtmp(testbox.to_s, hash) @es.update_tbox(testbox.to_s, hash) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 6e8d5d1..ff88d32 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -182,7 +182,7 @@ class Sched def get_job_boot(host, boot_type) queues = get_queues(host) job = get_job_from_queues(queues, host) - set_lifecycle(job, host) + set_lifecycle(job, host, queues) if job @es.set_job_content(job) -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: add parameters annotation
by Wang Yong 01 Apr '21

01 Apr '21

it's for choosing parameters clearly Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/rpmbuild-pkg | 1 + 1 file changed, 1 insertion(+) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 7a101b6a0..a3868d54a 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,6 +1,7 @@ #!/bin/bash # - upstream_repo # - compat_os +# Git repo jobs only use the above parameters, SRPMs jobs use all # - repo_name # - repo_addr -- 2.23.0

1 0

[PATCH v3 lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 01 Apr '21

01 Apr '21

[output] catch errors will be like this: { "openeuler_docker.wordpress_1.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql.message": [ "Error: Unable to find a match php-mysql" ], "openeuler_docker.tools_2.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng.message": [ "Error: Unable to find a match netsniff-ng" ], "openeuler_docker.yum.error.Unable-to-find-a-match.rpm-ostree": [ 1 ], ...... ...... } Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..5a9365938 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,103 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 01 Apr '21

01 Apr '21

[output] catch errors will be like this: { "openeuler_docker.wordpress_1.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.php-mysql.message": [ "Error: Unable to find a match php-mysql" ], "openeuler_docker.tools_2.build.fail": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng": [ 1 ], "openeuler_docker.yum.error.Unable-to-find-a-match.netsniff-ng.message": [ "Error: Unable to find a match netsniff-ng" ], "openeuler_docker.yum.error.Unable-to-find-a-match.rpm-ostree": [ 1 ], } Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 103 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 103 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..5a9365938 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,103 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

1 1

[PATCH compass-ci] doc: README: replace invalid url
by Liu Yinsi 01 Apr '21

01 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- README.en.md | 8 ++++---- README.zh.md | 8 ++++---- 2 files changed, 8 insertions(+), 8 deletions(-) diff --git a/README.en.md b/README.en.md index fa88305..7449696 100644 --- a/README.en.md +++ b/README.en.md @@ -12,7 +12,7 @@ Compass-CI monitors git repos of a large amount of open source software. Once Co **Login to the Test Environment** -The SSH is used to [log in to the test environment for commissioning](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…. +The SSH is used to [log in to the test environment for commissioning](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…. **Test Result Analysis** @@ -38,15 +38,15 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Manually Submitting a Test Task** -1. [Install the Compass-CI client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E…. +1. [Install the Compass-CI client](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/in…. 2. [Compile test cases and manually submit test tasks](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcas…. 3. You can [view](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/brow… and [compare](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/c… test results on the web page: https://compass-ci.openeuler.org/jobs **Logging in to the Test Environment** 1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. -2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. -3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. +2. Complete the environment configuration. +3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. ## Contributing to Compass-CI diff --git a/README.zh.md b/README.zh.md index 0e6cfb6..f897ae0 100644 --- a/README.zh.md +++ b/README.zh.md @@ -14,7 +14,7 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **调测环境登录** -使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… +使用 SSH [登录测试环境进行调测](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… **测试结果分析** @@ -41,15 +41,15 @@ Compass-CI 监控很多开源软件 git repos，一旦检测到代码更新， **手动提交测试任务** -1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9… +1. [安装 Compass-CI 客户端](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/insta… 2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca…, [手动提交测试任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… 3. 在网页中[查看](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/b… web: https://compass-ci.openeuler.org/jobs **登录测试环境** 1. 向 compass-ci-robot(a)qq.com 发送邮件[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual… -2. 根据邮件反馈内容[完成环境配置](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/… -3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… +2. 根据邮件反馈内容完成环境配置。 +3. 在测试任务中添加 sshd 字段，提交相应的任务，[登录测试环境](https://gitee.com/wu_fengguang/compass-ci/blob/master/d… ## Contributing to Compass-CI -- 2.23.0

1 0

[PATCH v3 compass-ci] container/ssh-r/start: rename variable name from vm_name to docker_name
by Wei Jihui 31 Mar '21

31 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..47a29eb 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

1 0

[PATCH lkp-tests] test/ansible_test: modify ansible code to output the module name
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] modify ansible code to output the name of the module for tasks execution. [How] print("{'action':'%s'}" % task_result.__dict__['_task_fields']['action']) {'action':'package'} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/tests/ansible_test b/tests/ansible_test index 56926bbed..8c3e68c59 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -58,7 +58,17 @@ ansible_playbook_run() EOF [ $? -eq 0 ] && { echo "Playbook has been generated!" - } + } + [ $(command -v ansible-playbook) ] || { + echo "ansible-playbook: command not found" + exit 1 + } + + sed -i "536a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + ansible-playbook "$ansible_path"/site.yml + + sed -i "550a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py ansible-playbook "$ansible_path"/site.yml if [ $? -eq 0 ]; then -- 2.23.0

1 1

[PATCH compass-ci] replace alpine mirrors
by Liu Yinsi 31 Mar '21

31 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/result-webdav/Dockerfile | 1 + container/srv-http/Dockerfile | 2 ++ container/upload-libvirt-xml/Dockerfile | 1 + 3 files changed, 4 insertions(+) diff --git a/container/result-webdav/Dockerfile b/container/result-webdav/Dockerfile index db48bcb..2edd2a7 100644 --- a/container/result-webdav/Dockerfile +++ b/container/result-webdav/Dockerfile @@ -5,6 +5,7 @@ FROM openresty/openresty:alpine +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add bash RUN mkdir -p /var/log/nginx/ RUN adduser lkp -u 1090 -g 1090 -D diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index b7d0677..4c010b9 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -18,6 +18,8 @@ FROM nginx:alpine ADD root / +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf diff --git a/container/upload-libvirt-xml/Dockerfile b/container/upload-libvirt-xml/Dockerfile index db48bcb..2edd2a7 100644 --- a/container/upload-libvirt-xml/Dockerfile +++ b/container/upload-libvirt-xml/Dockerfile @@ -5,6 +5,7 @@ FROM openresty/openresty:alpine +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add bash RUN mkdir -p /var/log/nginx/ RUN adduser lkp -u 1090 -g 1090 -D -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: add event time
by Wu Zhende 31 Mar '21

31 Mar '21

example: "submit_time": "2021-03-31T16:01:07+0800" "boot_time": "2021-03-31T16:01:33+0800" "close_time": "2021-03-31T16:02:47+0800" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 5 +++++ src/scheduler/close_job.cr | 5 ++++- src/scheduler/find_job_boot.cr | 1 + 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 5767f61..2d15aae 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -170,6 +170,11 @@ class Job set_sshr_info() set_queue() set_subqueue() + set_time("submit_time") + end + + def set_time(key) + self[key] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") end private def set_docker_os diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 746f70a..a112a38 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -15,6 +15,9 @@ class Sched job["job_state"] = job_state if job_state job["job_state"] = "complete" if job["job_state"] == "boot" + job.set_time("close_time") + @env.set "close_time", job["close_time"] + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -34,7 +37,7 @@ class Sched mq_msg = { "job_id" => @env.get?("job_id").to_s, "job_state" => "close", - "time" => get_time + "time" => @env.get?("close_time").to_s } spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 2596f83..6e8d5d1 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -92,6 +92,7 @@ class Sched if job job.update({"testbox" => testbox}) job.set_result_root + job.set_time("boot_time") @log.info(%({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state" : "set result root"})) set_id2job(job) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/git_mirror.rb: fix es update fail
by Li Yuanchao 31 Mar '21

31 Mar '21

As the es update to 7.11, some compatibility error occurs. [error msg] git-mirror: warning: 299 Elasticsearch-7.11.1-ff17057114c2199c9c1bbecc727003a907c0db7a "[types removal] Specifying types in doc ument update requests is deprecated, use the endpoint /{index}/_update/{id} instead." Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 7f36be3..1f4e5b5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -308,11 +308,8 @@ class MirrorMain def es_repo_update(git_repo) repo_info = { 'git_repo' => git_repo, 'url' => @git_info[git_repo]['url'] } repo_info = repo_info.merge(@fork_stat[git_repo]) - body = { - "doc": repo_info, - "doc_as_upsert": true - } - @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) + body = repo_info + @es_client.index(index: 'repo', type: '_doc', id: git_repo, body: body) end def update_stat_fetch(git_repo) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: fix no positive? method error
by Li Yuanchao 31 Mar '21

31 Mar '21

The result of query es changed, the result['hits']['total'] changes from a number to a hash, so the positive? method can not be used. Now use the count method which contained by es_client. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index c967c8b..7f36be3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -414,9 +414,9 @@ class MirrorMain new_refs_count: {} } query = { query: { match: { _id: git_repo } } } - result = @es_client.search(index: 'repo', body: query)['hits'] - return fork_stat unless result['total'].positive? + return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? + result = @es_client.search(index: 'repo', body: query)['hits'] fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0

[PATCH lkp-tests] add self-test.yaml
by Liu Yinsi 31 Mar '21

31 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- jobs/self-test.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/self-test.yaml diff --git a/jobs/self-test.yaml b/jobs/self-test.yaml new file mode 100644 index 000000000..ad2dcd2a2 --- /dev/null +++ b/jobs/self-test.yaml @@ -0,0 +1,6 @@ +suite: self-test +category: functional + +deploy-cci: +multi-qemu: +multi-docker: -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: add 8-testbox/multi-docker
by Liu Yinsi 31 Mar '21

31 Mar '21

For user locally deploy compass-ci, execute multi-docker in the background, it is convenient for users to submit jobs. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/8-testbox/multi-docker | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/8-testbox/multi-docker diff --git a/sparrow/8-testbox/multi-docker b/sparrow/8-testbox/multi-docker new file mode 100755 index 0000000..25cb82d --- /dev/null +++ b/sparrow/8-testbox/multi-docker @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +nohup $CCI_SRC/providers/multi-docker -n dc-8g -c 10 -q dc-8g.$(arch) & diff --git a/sparrow/install-tiny b/sparrow/install-tiny index e48bd25..40358a4 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -23,3 +23,4 @@ cd $CCI_SRC/sparrow || exit 5-build/os-ready 5-build/register-account 7-systemd/systemd-setup +8-testbox/multi-docker -- 2.23.0

1 0

[PATCH compass-ci] sparrow: add 8-testbox/multi-docker
by Liu Yinsi 31 Mar '21

31 Mar '21

For user locally deploy compass-ci, prepare to run some docker testbox, it is convenient for users to submit jobs. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/8-testbox/multi-docker | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/8-testbox/multi-docker diff --git a/sparrow/8-testbox/multi-docker b/sparrow/8-testbox/multi-docker new file mode 100755 index 0000000..94f6bde --- /dev/null +++ b/sparrow/8-testbox/multi-docker @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +$CCI_SRC/providers/multi-docker -n dc-8g -c 10 -q dc-8g.$(arch) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index e48bd25..40358a4 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -23,3 +23,4 @@ cd $CCI_SRC/sparrow || exit 5-build/os-ready 5-build/register-account 7-systemd/systemd-setup +8-testbox/multi-docker -- 2.23.0

2 1

[PATCH v2 compass-ci] doc: update sparrow README.md
by Liu Yinsi 31 Mar '21

31 Mar '21

Locally deploy compass-ci just for sparrow, if users want to use other services, prompt users to apply for account and borrow machine. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 54 +++++++++++++++-------------------------------- 1 file changed, 17 insertions(+), 37 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index d7b8e81..a448bb8 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -41,7 +41,7 @@ 3. 创建工作目录并克隆 compass-ci 项目代码 ```bash mkdir /c/ && ln -s /c/compass-ci /c/cci - git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/compass-ci.git ``` 4. 编辑setup.yaml配置用户名和邮箱 @@ -57,9 +57,14 @@ cd compass-ci/sparrow && ./install-tiny ``` -#### 提交测试任务前的准备 +#### 提交测试任务 +本文以/c/lkp-tests/jobs/目录下已有的通用测试用例iperf.yaml为例 +- 使环境变量生效 + ```bash + . /etc/profile.d/compass-ci + ``` -1. 测试环境是否可以提交job测试 +- 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash submit iperf.yaml ``` @@ -68,42 +73,17 @@ ``` submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - ``` - compass-ci搭建完毕，下面就可以开始进行测试了。 - -#### 提交测试任务到本地compass-ci -本文以测试用例iperf.yaml为例 - -1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - -2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… - -3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… - ```bash - submit iperf.yaml - ``` - -#### 运行测试任务并查看任务结果 - -1. 运行测试任务 - ```bash - cd /c/compass-ci/providers/ && ./my-qemu.sh + submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.2 ``` -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +- 查看任务结果 +任务结果文件会生成在本地/srv/result目录下，根据上一步骤显示出的job id可以查看对应任务的结果文件output(可使用tab键自动补全多级目录) ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 - cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/tcp-300/nolab.1 && cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/udp-300/nolab.2 && cat output ``` + + >**说明：** + >output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 - >**说明：** - >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - -# FAQ - -* 选择下载os rootfs - - 启动qemu测试机需要[os rootfs文件](http://api.compass-ci.openeuler.org:11304/os/)， - 一键部署默认下载os rootfs为openeuler aarch64 20.03，当部署完毕后，可在该目录下查看: /srv/os/openeuler/aarch64/20.03 - 如需要其他os rootfs，可使用该脚本下载: /c/compass-ci/sbin/download-rootfs +体验更多服务例如[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md)、[调测环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/log-in-machine-debug.md)、[测试结果分析](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compare-results.zh.md)等，请[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.zh.md)、[申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/borrow-machine.zh.md)。 -- 2.23.0

1 0

[PATCH compass-ci] doc: update sparrow README.md
by Liu Yinsi 31 Mar '21

31 Mar '21

Locally deploy compass-ci just for sparrow, if users want to use other services, prompt users to apply for account and borrow machine. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 50 ++++++++++++----------------------------------- 1 file changed, 13 insertions(+), 37 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index d7b8e81..b161877 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -41,7 +41,7 @@ 3. 创建工作目录并克隆 compass-ci 项目代码 ```bash mkdir /c/ && ln -s /c/compass-ci /c/cci - git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/compass-ci.git ``` 4. 编辑setup.yaml配置用户名和邮箱 @@ -57,9 +57,10 @@ cd compass-ci/sparrow && ./install-tiny ``` -#### 提交测试任务前的准备 +#### 提交测试任务 +本文以/c/lkp-tests/jobs/目录下已有的通用测试用例iperf.yaml为例 -1. 测试环境是否可以提交job测试 +1. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash submit iperf.yaml ``` @@ -68,42 +69,17 @@ ``` submit_id=bf5e7ad7-839d-48ec-a033-23281323c750 submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.1 - ``` - compass-ci搭建完毕，下面就可以开始进行测试了。 - -#### 提交测试任务到本地compass-ci -本文以测试用例iperf.yaml为例 - -1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… - -2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… - -3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… - ```bash - submit iperf.yaml + submit /c/lkp-tests/jobs/iperf.yaml, got job id=nolab.2 ``` -#### 运行测试任务并查看任务结果 - -1. 运行测试任务 - ```bash - cd /c/compass-ci/providers/ && ./my-qemu.sh - ``` - -2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/lab-… [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +2. 查看任务结果 +任务结果文件会生成在本地/srv/result目录下，根据上一步骤显示出的job id可以查看对应任务的结果文件output(可使用tab键自动补全多级目录) ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 - cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/tcp-300/nolab.1 && cat output + cd /srv/result/iperf/2021-03-18/dc-8g/centos-7-aarch64/udp-300/nolab.2 && cat output ``` + + >**说明：** + >output 文件记录了用例的执行过程，文件最后部分一般会有 check_exit_code 这个状态码，非 0 代表测试用例错误。 - >**说明：** - >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… - -# FAQ - -* 选择下载os rootfs - - 启动qemu测试机需要[os rootfs文件](http://api.compass-ci.openeuler.org:11304/os/)， - 一键部署默认下载os rootfs为openeuler aarch64 20.03，当部署完毕后，可在该目录下查看: /srv/os/openeuler/aarch64/20.03 - 如需要其他os rootfs，可使用该脚本下载: /c/compass-ci/sbin/download-rootfs +体验更多服务例如[自动化测试](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md)、[调测环境登录](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/log-in-machine-debug.md)、[测试结果分析](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/compare-results.zh.md)等，请[申请账号](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.zh.md)、[申请测试机](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/borrow-machine.zh.md)。 -- 2.23.0

1 1

[PATCH compass-ci 1/2] lib/stats_wrapper.rb: fix TypeError
by Lu Weitao 31 Mar '21

31 Mar '21

[Why] linjiaxin@crystal ~/crystal.116494% ~/compass-ci/sbin/result2stats ./ Traceback (most recent call last): 7: from /home/linjiaxin/compass-ci/sbin/result2stats:37:in `<main>' 6: from /home/linjiaxin/compass-ci/lib/stats.rb:47:in `extract_stats' 5: from /usr/share/ruby/set.rb:338:in `each' 4: from /usr/share/ruby/set.rb:338:in `each_key' 3: from /home/linjiaxin/compass-ci/lib/stats.rb:49:in `block in extract_stats' 2: from /home/linjiaxin/compass-ci/lib/stats_wrapper.rb:61:in `wrapper_func' 1: from /home/linjiaxin/compass-ci/lib/stats_wrapper.rb:287:in `delete_log_package' /home/linjiaxin/compass-ci/lib/stats_wrapper.rb:287:in `exist?': no implicit conversion of nil into String (TypeError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index faa81df..6a08797 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -284,7 +284,7 @@ module StatsWrapper # delete temporarily extracted dmesg above File.delete(@dmesg_log) if File.exist?("#{(a)dmesg_log}.xz") - File.delete(@tmpfile) if File.exist?(@tmpfile) + File.delete(@tmpfile) if @tmpfile && File.exist?(@tmpfile) end end -- 2.23.0

2 1

[PATCH compass-ci 2/2] lib/matrix2.rb: fix NoMethodError
by Lu Weitao 31 Mar '21

31 Mar '21

[Why] linjiaxin@crystal ~/crystal.116494% ~/compass-ci/sbin/result2stats ./ Traceback (most recent call last): 7: from /home/linjiaxin/compass-ci/sbin/result2stats:38:in `<main>' 6: from /home/linjiaxin/compass-ci/lib/matrix2.rb:52:in `create_stats' 5: from /home/linjiaxin/compass-ci/lib/matrix2.rb:52:in `each' 4: from /home/linjiaxin/compass-ci/lib/matrix2.rb:58:in `block in create_stats' 3: from /home/linjiaxin/compass-ci/lib/matrix2.rb:30:in `extract_pre_result' 2: from /home/linjiaxin/compass-ci/lib/matrix2.rb:30:in `each' 1: from /home/linjiaxin/compass-ci/lib/matrix2.rb:33:in `block in extract_pre_result' /home/linjiaxin/compass-ci/lib/matrix2.rb:22:in `set_pre_value': undefined method `/' for #<String:0x0000aaabcdd1a140> (NoMethodError) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 5e0ad21..12245a8 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -10,7 +10,7 @@ require "#{LKP_SRC}/lib/matrix" require_relative './params_group.rb' def set_pre_value(item, value, sample_size) - return value if value.is_a?(String) + return value if item.end_with?('.message', '.log') if value.size == 1 value[0] -- 2.23.0

2 1

[PATCH lkp-tests 2/2] stats: add build-pkg.rb to generate error_id with message
by Lin Jiaxin 31 Mar '21

31 Mar '21

output like this: "build-pkg.include/common/ist.h:warning:unused-parameter‘count’[-Wunused-parameter].message": [ "include/common/ist.h:488:74: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str(char *dst, const struct ist src, size_t count)\n ^~~~~\n", "include/common/ist.h:533:77: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str_lc(char *dst, const struct ist src, size_t count )\n ^~~~~\n", "include/common/ist.h:578:77: warning: unused parameter ‘count’ [-Wunused-parameter]\n static inline struct ist ist2str_uc(char *dst, const struct ist src, size_t count )\n ^~~~~\n" ] Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg.rb | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 stats/build-pkg.rb diff --git a/stats/build-pkg.rb b/stats/build-pkg.rb new file mode 100755 index 00000000..a987e318 --- /dev/null +++ b/stats/build-pkg.rb @@ -0,0 +1,73 @@ +#!/usr/bin/env ruby + +LKP_SRC ||= ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require 'set' +require "#{LKP_SRC}/lib/common" + +class ErrorMessages + def initialize + @seqno = '' + @in_stderr = false + @error_line = '' + @error_details = '' + @error_message = {} + @error_messages = Hash.new { |h, k| h[k] = Set.new } + end + + def obtain_error_messages(log_lines) + log_lines.each do |line| + next if extract_error_message(line) + next unless @in_stderr + next unless @error_message['error_line'] =~ /(error|warning):[^:]/i + + error_id = build_pkg_error_id(@error_message['error_line']) + @error_messages[error_id] << @error_message['error_line'] + @error_message['error_details'] + end + return @error_messages + end + + private + + def extract_error_message(line) + if line =~ /^ / + @error_details += line + else + update_error_message(line) + case line + when /^## ______________([0-9.]+):stderr$/ + @in_stderr = true + @seqno = $1 + return true + + when /^## ______________#{@seqno}:enderr$/ + @in_stderr = false + @seqno = '' + add_error_message + end + end + end + + def update_error_message(line) + @error_message['error_line'] = @error_line + @error_message['error_details'] = @error_details + @error_line = line + @error_details = '' + end + + def add_error_message + return unless @error_message['error_line'] =~ /(error|warning):[^:]/i + + error_id = build_pkg_error_id(@error_message['error_line']) + @error_messages[error_id] << @error_message['error_line'] + @error_message['error_details'] + end +end + +def build_pkg(log_lines) + message = {} + ErrorMessages.new.obtain_error_messages(log_lines).each do |k, v| + message["#{k}.message"] = v.to_a + end + + return message +end -- 2.23.0

1 0

[PATCH lkp-tests 1/2] stats/build-pkg: don't initialize LKP_SRC repeatedly
by Lin Jiaxin 31 Mar '21

31 Mar '21

error message: /home/linjiaxin/lkp-tests/stats/build-pkg.rb:3: warning: already initialized constant LKP_SRC Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- stats/build-pkg | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/stats/build-pkg b/stats/build-pkg index 764594b7..0caf1a73 100755 --- a/stats/build-pkg +++ b/stats/build-pkg @@ -1,6 +1,6 @@ #!/usr/bin/env ruby -LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) +LKP_SRC ||= ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) require 'set' require "#{LKP_SRC}/lib/common" -- 2.23.0

1 0

[PATCH compass-ci 2/4] src/scheduler/kernel_params.cr: add param pv_device into dracut initrd for local mount
by Xu Xijian 31 Mar '21

31 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/scheduler/kernel_params.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 8ad5029..9e2f58b 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -9,7 +9,8 @@ class Job os_info = "#{os}_#{os_arch}_#{os_version}" use_root_partition = "/dev/mapper/os-#{os_info}_#{src_lv_suffix}" if @hash["src_lv_suffix"]? != nil save_root_partition = "/dev/mapper/os-#{os_info}_#{boot_lv_suffix}" if @hash["boot_lv_suffix"]? != nil - return "#{common_params} local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} rw" + pv_device = "#{self.pv_device}" if @hash["pv_device"]? != nil + return "#{common_params} local use_root_partition=#{use_root_partition} save_root_partition=#{save_root_partition} pv_device=#{pv_device} rw" end private def kernel_custom_params -- 2.23.0

2 2

[PATCH compass-ci 4/4] container/dracut-initrd: enable creating pv and vg for lv to support local mount in HW machine
by Xu Xijian 31 Mar '21

31 Mar '21

[why] If a HW machine has no physical volume or volume group for logical volume, we should support to create them by job param, this param named as pv_device(default empty and do nothing). If a usable $pv_device is given, dracut will check the device and ensure pv and vg. Usage: submit job.yaml ... pv_device=/dev/sda Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../dracut-initrd/bin/set-local-sysroot.sh | 27 +++++++++++++++++++ 1 file changed, 27 insertions(+) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 835bdd2..2a07270 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -14,9 +14,36 @@ analyse_kernel_cmdline_params() { sync_src_lv() { local src_lv="$1" + local vg_name="os" [ -e "$src_lv" ] && return + # need create volume group, usually in first use of this machine. $pv_device e.g. /dev/sda + pv_device="$(getarg pv_device=)" + [ -n "$pv_device" ] && { + [ -b "$pv_device" ] || { + echo "warn dracut: FATAL: device not found: $pv_device, reboot" + reboot + } + + # ensure the physical disk has been initialized as physical volume + real_pv_device="$(lvm pvs | grep -w $pv_device | awk '{print $1}')" + [ "$real_pv_device" = "$pv_device" ] || { + lvm pvcreate "$pv_device" || reboot + } + + # ensure the volume group $vg_name exists + real_vg_name="$(lvm pvs | grep -w $vg_name | awk '{print $2}')" + [ "$real_vg_name" = "$vg_name" ] || { + lvm vgcreate "$vg_name" "$pv_device" || reboot + } + } + + lvm vgs "$vg_name" || { + echo "warn dracut: FATAL: vg os not found, reboot" + reboot + } + # create logical volume src_lv_devname="$(basename $src_lv)" lvm lvcreate -y -L 10G --name "${src_lv_devname#os-}" os -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] container/es: upgrade version
by Wu Zhende 31 Mar '21

31 Mar '21

Upgrade to 7.11.1. Matches the version of kibana so that data can be displayed on kibana. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/es/Dockerfile | 48 ++++++++++++-------------------------- container/es/build | 4 ++-- container/es/start | 2 +- container/logging-es/build | 8 +++---- container/logging-es/start | 2 +- 5 files changed, 23 insertions(+), 41 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index ed02490..44d87ee 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -1,41 +1,24 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786 ARG MEMORY -RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories - -RUN apk add --no-cache elasticsearch curl - -RUN rm -rf /etc/init.d/elasticsearch \ - && rm -rf /usr/share/java/elasticsearch/plugins \ - && mv /usr/share/java/elasticsearch /usr/share/es \ - && echo "===> Creating Elasticsearch Paths..." \ - && for path in \ - /srv/es \ - /usr/share/es/logs \ - /usr/share/es/config \ - /usr/share/es/config/scripts \ - /usr/share/es/tmp \ - /usr/share/es/plugins \ - ; do \ - mkdir -p "$path"; \ - done \ - && cp /etc/elasticsearch/*.* /usr/share/es/config \ - && chown -R 1090:1090 /usr/share/es \ - && chown -R 1090:1090 /srv/es; - -RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options - -WORKDIR /usr/share/es - -ENV PATH /usr/share/es/bin:$PATH -ENV ES_TMPDIR /usr/share/es/tmp +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options + +RUN mkdir /usr/share/elasticsearch/tmp && \ + chown -R 1090:1090 /usr/share/elasticsearch + +WORKDIR /usr/share/elasticsearch + +ENV PATH /usr/share/elasticsearch/bin:$PATH +ENV ES_TMPDIR /usr/share/elasticsearch/tmp VOLUME ["/srv/es"] @@ -43,4 +26,3 @@ EXPOSE 9200 9300 USER 1090 CMD ["elasticsearch"] - diff --git a/container/es/build b/container/es/build index c7d7115..db5145f 100755 --- a/container/es/build +++ b/container/es/build @@ -5,8 +5,8 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "es643b:alpine311" +docker_skip_rebuild "es:7.11.1" available_memory = get_available_memory -system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." +system "docker build -t es:7.11.1 --build-arg MEMORY=#{available_memory} --network=host ." diff --git a/container/es/start b/container/es/start index 67d6531..3aa9525 100755 --- a/container/es/start +++ b/container/es/start @@ -15,7 +15,7 @@ cmd=( -v /srv/es:/srv/es -v /etc/localtime:/etc/localtime:ro --name es-server01 - es643b:alpine311 + es:7.11.1 ) "${cmd[@]}" diff --git a/container/logging-es/build b/container/logging-es/build index 3be841a..b50830e 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -5,15 +5,15 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "logging-es:7.6.2" +docker_skip_rebuild "logging-es:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', - 'x86_64' => 'elasticsearch:7.6.2' + 'aarch64' => 'elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786', + 'x86_64' => 'elasticsearch:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] available_memory = get_available_memory -system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." +system "docker build -t logging-es:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..05ac46f 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -15,7 +15,7 @@ cmd=( -p 9302:9300 -v /srv/es/logging-es:/srv/es/logging-es --name logging-es - logging-es:7.6.2 + logging-es:7.11.1 ) "${cmd[@]}" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/kibana: upgrade and addition
by Wu Zhende 31 Mar '21

31 Mar '21

1. upgrade from 7.6.2 to 7.11.1 2. add a new kibana for data. one display log and one display data 3. delete logtrail plugin because it don't support in new kibana version Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana-logging/Dockerfile | 12 ++++++++ container/kibana-logging/build | 16 ++++++++++ container/kibana-logging/start | 31 +++++++++++++++++++ container/kibana-logging/start-depends | 1 + container/kibana/Dockerfile | 10 ------- container/kibana/build | 8 ++--- container/kibana/logtrail.json | 41 -------------------------- container/kibana/start | 14 ++++----- container/kibana/start-depends | 2 +- 9 files changed, 72 insertions(+), 63 deletions(-) create mode 100644 container/kibana-logging/Dockerfile create mode 100755 container/kibana-logging/build create mode 100755 container/kibana-logging/start create mode 100755 container/kibana-logging/start-depends delete mode 100644 container/kibana/logtrail.json diff --git a/container/kibana-logging/Dockerfile b/container/kibana-logging/Dockerfile new file mode 100644 index 0000000..970eb5a --- /dev/null +++ b/container/kibana-logging/Dockerfile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} + +# docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 + +MAINTAINER Wu Zhende <wuzhende666(a)163.com> + +RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml diff --git a/container/kibana-logging/build b/container/kibana-logging/build new file mode 100755 index 0000000..9c75aeb --- /dev/null +++ b/container/kibana-logging/build @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig' + +docker_skip_rebuild "kibana:7.11.1" + +BASE_IMAGE_DICT = { + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana-logging/start b/container/kibana-logging/start new file mode 100755 index 0000000..f2169a6 --- /dev/null +++ b/container/kibana-logging/start @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' +LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' + +docker_rm 'kibana-logging' + +cmd = %W[ + docker run + --restart=always + --name kibana-logging + -v /etc/localtime:/etc/localtime:ro + -d + -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} + -p 20000:5601 + kibana:7.11.1 +] + +system(*cmd) diff --git a/container/kibana-logging/start-depends b/container/kibana-logging/start-depends new file mode 100755 index 0000000..66c1996 --- /dev/null +++ b/container/kibana-logging/start-depends @@ -0,0 +1 @@ +logging-es diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 60b889d..970eb5a 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -10,13 +10,3 @@ FROM ${BASE_IMAGE} MAINTAINER Wu Zhende <wuzhende666(a)163.com> RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml - -USER root -RUN yum -y install wget \ - && wget https://github.com/sivasamyk/logtrail/releases/download/v0.1.31/logtrail-7.… -O /logtrail-7.6.2-0.1.31.zip - -USER 1090 -RUN ./bin/kibana-plugin install file:///logtrail-7.6.2-0.1.31.zip \ - && rm -rf /tmp/logtrail-7.6.2-0.1.31.zip - -COPY --chown=1090:1090 logtrail.json /usr/share/kibana/plugins/logtrail/logtrail.json diff --git a/container/kibana/build b/container/kibana/build index 9a1fb6d..9c75aeb 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -5,12 +5,12 @@ require_relative '../defconfig' -docker_skip_rebuild "kibana:7.6.2" +docker_skip_rebuild "kibana:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', - 'x86_64' => 'kibana:7.6.2' }.freeze + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json deleted file mode 100644 index c63f2ac..0000000 --- a/container/kibana/logtrail.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "version": 2, - "index_patterns": [{ - "es": { - "default_index": "*", - "allow_url_parameter": false, - "timezone": "CST" - }, - "tail_interval_in_seconds": 10, - "es_index_time_offset_in_seconds": 0, - "display_timezone": "CST", - "display_timestamp_format": "YYYY MMM DD HH:mm:ss", - "max_buckets": 500, - "nested_objects": false, - "default_time_range_in_days": 30, - "max_hosts": 100, - "max_events_to_keep_in_viewer": 10000, - "default_search": "", - "fields": { - "mapping": { - "timestamp": "time", - "display_timestamp" : "time", - "hostname": "container_name", - "program": "tags", - "message": "log" - }, - "message_format": "{{{log}}}", - "keyword_suffix": "keyword" - }, - "color_mapping": { - "field": "level", - "mapping": { - "ERROR": "#FF0000", - "WARN": "#FFEF96", - "DEBUG": "#B5E7A0", - "TRACE": "#CFE0E8", - "INFO": "#339999" - } - } - }] -} diff --git a/container/kibana/start b/container/kibana/start index 26a4944..63cb8f8 100755 --- a/container/kibana/start +++ b/container/kibana/start @@ -7,13 +7,13 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - LOGGING_ES_HOST - LOGGING_ES_PORT + ES_HOST + ES_PORT ] defaults = relevant_defaults(names) -LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' -LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' +ES_HOST = defaults['ES_HOST'] || '172.17.0.1' +ES_PORT = defaults['ES_PORT'] || '9200' docker_rm 'kibana' @@ -23,9 +23,9 @@ cmd = %W[ --name kibana -v /etc/localtime:/etc/localtime:ro -d - -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} - -p 20000:5601 - kibana:7.6.2 + -e ELASTICSEARCH_HOSTS=http://#{ES_HOST}:#{ES_PORT} + -p 20017:5601 + kibana:7.11.1 ] system(*cmd) diff --git a/container/kibana/start-depends b/container/kibana/start-depends index 66c1996..8357fca 100755 --- a/container/kibana/start-depends +++ b/container/kibana/start-depends @@ -1 +1 @@ -logging-es +es -- 2.23.0

1 0

[PATCH compass-ci] lib/git_mirror.rb: fix no positive? method error
by Li Yuanchao 31 Mar '21

31 Mar '21

The result of query es changed, the result['hits']['total'] changes from a number to a hash, so the positive? method can not be used. Now use the count method which contained by es_client. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index c967c8b..7f36be3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -414,9 +414,9 @@ class MirrorMain new_refs_count: {} } query = { query: { match: { _id: git_repo } } } - result = @es_client.search(index: 'repo', body: query)['hits'] - return fork_stat unless result['total'].positive? + return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? + result = @es_client.search(index: 'repo', body: query)['hits'] fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/es: upgrade version
by Wu Zhende 31 Mar '21

31 Mar '21

Upgrade to 7.11.1. Matches the version of kibana so that data can be displayed on kibana. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/es/Dockerfile | 48 ++++++++++++-------------------------- container/es/build | 4 ++-- container/es/start | 2 +- container/logging-es/build | 8 +++---- container/logging-es/start | 2 +- 5 files changed, 23 insertions(+), 41 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index ed02490..44d87ee 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -1,41 +1,24 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786 ARG MEMORY -RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories - -RUN apk add --no-cache elasticsearch curl - -RUN rm -rf /etc/init.d/elasticsearch \ - && rm -rf /usr/share/java/elasticsearch/plugins \ - && mv /usr/share/java/elasticsearch /usr/share/es \ - && echo "===> Creating Elasticsearch Paths..." \ - && for path in \ - /srv/es \ - /usr/share/es/logs \ - /usr/share/es/config \ - /usr/share/es/config/scripts \ - /usr/share/es/tmp \ - /usr/share/es/plugins \ - ; do \ - mkdir -p "$path"; \ - done \ - && cp /etc/elasticsearch/*.* /usr/share/es/config \ - && chown -R 1090:1090 /usr/share/es \ - && chown -R 1090:1090 /srv/es; - -RUN sed -i 's:#path.data\: /path/to/data:path.data\: /srv/es:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/es/config/elasticsearch.yml; -RUN sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/es/config/jvm.options -RUN sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/es/config/jvm.options - -WORKDIR /usr/share/es - -ENV PATH /usr/share/es/bin:$PATH -ENV ES_TMPDIR /usr/share/es/tmp +RUN sed -i 's:#network.host\: _site_:network.host\: 0.0.0.0:' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a path.data: /srv/es' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a node.name: node-1' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i '$a cluster.initial_master_nodes: ["node-1"]' /usr/share/elasticsearch/config/elasticsearch.yml && \ + sed -i "s/-Xms256m/-Xms${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options && \ + sed -i "s/-Xmx256m/-Xmx${MEMORY}m/g" /usr/share/elasticsearch/config/jvm.options + +RUN mkdir /usr/share/elasticsearch/tmp && \ + chown -R 1090:1090 /usr/share/elasticsearch + +WORKDIR /usr/share/elasticsearch + +ENV PATH /usr/share/elasticsearch/bin:$PATH +ENV ES_TMPDIR /usr/share/elasticsearch/tmp VOLUME ["/srv/es"] @@ -43,4 +26,3 @@ EXPOSE 9200 9300 USER 1090 CMD ["elasticsearch"] - diff --git a/container/es/build b/container/es/build index c7d7115..db5145f 100755 --- a/container/es/build +++ b/container/es/build @@ -5,8 +5,8 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "es643b:alpine311" +docker_skip_rebuild "es:7.11.1" available_memory = get_available_memory -system "docker build -t es643b:alpine311 --build-arg MEMORY=#{available_memory} --network=host ." +system "docker build -t es:7.11.1 --build-arg MEMORY=#{available_memory} --network=host ." diff --git a/container/es/start b/container/es/start index 67d6531..3aa9525 100755 --- a/container/es/start +++ b/container/es/start @@ -15,7 +15,7 @@ cmd=( -v /srv/es:/srv/es -v /etc/localtime:/etc/localtime:ro --name es-server01 - es643b:alpine311 + es:7.11.1 ) "${cmd[@]}" diff --git a/container/logging-es/build b/container/logging-es/build index 3be841a..b50830e 100755 --- a/container/logging-es/build +++ b/container/logging-es/build @@ -5,15 +5,15 @@ require_relative '../defconfig.rb' -docker_skip_rebuild "logging-es:7.6.2" +docker_skip_rebuild "logging-es:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/elasticsearch-oss-arm64:7.6.2', - 'x86_64' => 'elasticsearch:7.6.2' + 'aarch64' => 'elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786', + 'x86_64' => 'elasticsearch:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] available_memory = get_available_memory -system "docker build -t logging-es:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." +system "docker build -t logging-es:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg MEMORY=#{available_memory} ." diff --git a/container/logging-es/start b/container/logging-es/start index 9be7a5b..05ac46f 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -15,7 +15,7 @@ cmd=( -p 9302:9300 -v /srv/es/logging-es:/srv/es/logging-es --name logging-es - logging-es:7.6.2 + logging-es:7.11.1 ) "${cmd[@]}" -- 2.23.0

2 1

[PATCH lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 31 Mar '21

31 Mar '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 102 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 102 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..c4a2cd691 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,102 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + next + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_test: optimize the output of error_id
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 99 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 83 insertions(+), 16 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..cadc11786 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,107 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + # when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end + +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case i + when /(.*)/ + output_error $1 + end + end +end + while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + when /(FAILED!|failed:).*=>(.*)/ - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? - ansible_failed_json = JSON.parse @ansible_failed_info + ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

1 0

[PATCH compass-ci 3/4] doc/job/os_mount.md: add the check pv_device step into document
by Xu Xijian 31 Mar '21

31 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- doc/job/os_mount.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/doc/job/os_mount.md b/doc/job/os_mount.md index eab95ed..b9b8308 100644 --- a/doc/job/os_mount.md +++ b/doc/job/os_mount.md @@ -47,6 +47,7 @@ The brief flow is as follows: ${boot_lv} -- boot logical volume: - will boot from this lv. - boot_lv=/dev/mapper/os-${os}_${os_arch}_${os_version} + - check whether ${pv_device} is given, if so, create physical volume(pv for short) and volume group(vg for short) on ${pv_device}. - if ${src_lv} not exists: create it, and rsync the rootfs from cluster nfs server. - if ${boot_lv} exists: delete it. - create ${boot_lv} as the snapshot of ${src_lv}. @@ -57,6 +58,7 @@ The brief flow is as follows: ## persistent rootfs data When you need to persist the rootfs data of a job, and use it in the subsequent job(s), two fields in `kernel_custom_params` will help you: 'src_lv_suffix', 'boot_lv_suffix'. +If you want to run on a brand new machine, you should use 'pv_device' to assign a disk device to create pv and vg. The brief flow is as follows: @@ -65,6 +67,7 @@ The brief flow is as follows: 2. initrd stage: use_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${src_lv_suffix} save_root_partition seems like /dev/mapper/os-${os}_${os_arch}_${os_version}_${boot_lv_suffix} + check and ensure pv and vg exist for lv. - firstly, we need two logical volume: ${src_lv} -- src logical volume: - if have ${use_root_partition}, src_lv=${use_root_partition} @@ -95,6 +98,8 @@ Demo usage: data of job-20210218.yaml. Then you need add the follow field in your job-20210219.yaml: kernel_custom_params: src_lv_suffix=zhangsan_local_for_iperf_20210218 + - if you want to create pv and vg on /dev/sda, you can use like this: + kernel_custom_params: pv_device=/dev/sda ... ``` Notes: @@ -148,6 +153,9 @@ Notes: lvdisplay ${src_lv} > /dev/null && return + # need create volume group, usually in first use of this machine. $pv_device e.g. /dev/sda + [ -n "$pv_device" ] && { do some check and create pv and vg } + # create logical volume lvcreate --size 10G --name $(basename ${src_lv}) os || exit -- 2.23.0

1 0

[PATCH compass-ci 1/4] src/lib/job.cr: add initrd param pv_device to scheduler
by Xu Xijian 31 Mar '21

31 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 5767f61..3ba991d 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -90,6 +90,7 @@ class Job linux_vmlinuz_path src_lv_suffix boot_lv_suffix + pv_device ) macro method_missing(call) -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_test:
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 99 ++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 83 insertions(+), 16 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..cadc11786 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,107 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + # when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end + +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case i + when /(.*)/ + output_error $1 + end + end +end + while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + when /(FAILED!|failed:).*=>(.*)/ - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? - ansible_failed_json = JSON.parse @ansible_failed_info + ansible_failed_json = JSON.parse @ansible_failed_info - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

1 1

[PATCH lab-z9] hosts: add rootfs_disk field
by Yu Chuan 31 Mar '21

31 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- hosts/taishan200-2280-2s48p-256g--a101 | 3 ++- hosts/taishan200-2280-2s48p-256g--a103 | 3 ++- hosts/taishan200-2280-2s48p-256g--a104 | 3 ++- hosts/taishan200-2280-2s48p-256g--a106 | 3 ++- hosts/taishan200-2280-2s48p-256g--a107 | 3 ++- hosts/taishan200-2280-2s48p-256g--a20 | 3 ++- hosts/taishan200-2280-2s48p-256g--a63 | 3 ++- hosts/taishan200-2280-2s48p-256g--a64 | 3 ++- hosts/taishan200-2280-2s48p-256g--a65 | 3 ++- hosts/taishan200-2280-2s48p-512g--a102 | 3 ++- hosts/taishan200-2280-2s48p-512g--a105 | 3 ++- hosts/taishan200-2280-2s48p-512g--a95 | 3 ++- hosts/taishan200-2280-2s64p-256g--a110 | 3 ++- hosts/taishan200-2280-2s64p-256g--a112 | 3 ++- hosts/taishan200-2280-2s64p-256g--a113 | 3 ++- hosts/taishan200-2280-2s64p-256g--a114 | 3 ++- hosts/taishan200-2280-2s64p-256g--a116 | 3 ++- hosts/taishan200-2280-2s64p-256g--a117 | 3 ++- hosts/taishan200-2280-2s64p-256g--a118 | 3 ++- hosts/taishan200-2280-2s64p-256g--a119 | 3 ++- hosts/taishan200-2280-2s64p-256g--a121 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a123 | 3 ++- hosts/taishan200-2280-2s64p-256g--a124 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a5 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a6 | 4 ++-- hosts/taishan200-2280-2s64p-256g--a7 | 4 ++-- 26 files changed, 52 insertions(+), 31 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g--a101 b/hosts/taishan200-2280-2s48p-256g--a101 index f8a042942dd7..9d40f98e6434 100644 --- a/hosts/taishan200-2280-2s48p-256g--a101 +++ b/hosts/taishan200-2280-2s48p-256g--a101 @@ -2,11 +2,12 @@ arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 256G -nr_hdd_partitions: 3 +nr_hdd_partitions: 2 nr_ssd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399b89192e1 - /dev/disk/by-id/scsi-350000399b88bf66d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bef85 ssd_partitions: - /dev/disk/by-id/ata-INTEL_SSDSC2KB480G8_BTYF94730FF4480BGN diff --git a/hosts/taishan200-2280-2s48p-256g--a103 b/hosts/taishan200-2280-2s48p-256g--a103 index 9fcc7187a8a3..ca26e20ec5e4 100644 --- a/hosts/taishan200-2280-2s48p-256g--a103 +++ b/hosts/taishan200-2280-2s48p-256g--a103 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bdc69 - /dev/disk/by-id/scsi-350000399b88bdd85 - /dev/disk/by-id/scsi-350000399b88bdc4d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88bdc7d mac_addr: - 44:67:47:c9:d8:b8 diff --git a/hosts/taishan200-2280-2s48p-256g--a104 b/hosts/taishan200-2280-2s48p-256g--a104 index 41cdbb052709..d0169640f279 100644 --- a/hosts/taishan200-2280-2s48p-256g--a104 +++ b/hosts/taishan200-2280-2s48p-256g--a104 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8892431 - /dev/disk/by-id/scsi-350000399b8892f81 - /dev/disk/by-id/scsi-350000399b8891c49 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88930d1 mac_addr: - 44:67:47:d7:6d:f0 diff --git a/hosts/taishan200-2280-2s48p-256g--a106 b/hosts/taishan200-2280-2s48p-256g--a106 index 95b2f0500c4c..de0f418250fd 100644 --- a/hosts/taishan200-2280-2s48p-256g--a106 +++ b/hosts/taishan200-2280-2s48p-256g--a106 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b883bf99 - /dev/disk/by-id/scsi-350000399b883ba89 - /dev/disk/by-id/scsi-350000399b883b94d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b883b981 mac_addr: - 44:67:47:86:1c:30 diff --git a/hosts/taishan200-2280-2s48p-256g--a107 b/hosts/taishan200-2280-2s48p-256g--a107 index 31b8dc6cdf90..f8506b22e38e 100644 --- a/hosts/taishan200-2280-2s48p-256g--a107 +++ b/hosts/taishan200-2280-2s48p-256g--a107 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-256g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b8917c39 - /dev/disk/by-id/scsi-350000399b8918005 - /dev/disk/by-id/scsi-350000399b8900a4d +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8900799 mac_addr: - 84:46:fe:73:a2:71 diff --git a/hosts/taishan200-2280-2s48p-256g--a20 b/hosts/taishan200-2280-2s48p-256g--a20 index 21bf8e1fcd9d..6d50eead2f31 100644 --- a/hosts/taishan200-2280-2s48p-256g--a20 +++ b/hosts/taishan200-2280-2s48p-256g--a20 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b888f2dd - /dev/disk/by-id/scsi-350000399b888f705 - /dev/disk/by-id/scsi-350000399b88906f5 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8890141 mac_addr: - 84:46:fe:3d:a7:d5 diff --git a/hosts/taishan200-2280-2s48p-256g--a63 b/hosts/taishan200-2280-2s48p-256g--a63 index 0b3c938372f3..fea8ccb98892 100644 --- a/hosts/taishan200-2280-2s48p-256g--a63 +++ b/hosts/taishan200-2280-2s48p-256g--a63 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b0431 - /dev/disk/by-id/scsi-350000399886adc71 - /dev/disk/by-id/scsi-350000399886b66f5 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b6b75 mac_addr: - 44:67:47:c9:e5:44 diff --git a/hosts/taishan200-2280-2s48p-256g--a64 b/hosts/taishan200-2280-2s48p-256g--a64 index 7cd0c4813ac6..b1fc7d2a25d5 100644 --- a/hosts/taishan200-2280-2s48p-256g--a64 +++ b/hosts/taishan200-2280-2s48p-256g--a64 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b2fa1 - /dev/disk/by-id/scsi-350000399b88b4339 - /dev/disk/by-id/scsi-350000399b88b4329 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88aefc1 mac_addr: - 44:67:47:c9:ea:b2 diff --git a/hosts/taishan200-2280-2s48p-256g--a65 b/hosts/taishan200-2280-2s48p-256g--a65 index 88441cc536d9..2bafc158a6a0 100644 --- a/hosts/taishan200-2280-2s48p-256g--a65 +++ b/hosts/taishan200-2280-2s48p-256g--a65 @@ -1,9 +1,10 @@ memory: 256G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88b4509 - /dev/disk/by-id/scsi-350000399b88b30dd - /dev/disk/by-id/scsi-350000399b88b4621 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b88b4425 mac_addr: - 44:67:47:c9:e9:90 diff --git a/hosts/taishan200-2280-2s48p-512g--a102 b/hosts/taishan200-2280-2s48p-512g--a102 index e1d3f10b8adb..ca88543ffc9b 100644 --- a/hosts/taishan200-2280-2s48p-512g--a102 +++ b/hosts/taishan200-2280-2s48p-512g--a102 @@ -2,10 +2,11 @@ arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 nr_ssd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b88bfbc9 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b8919395 ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH1T9HMLT-00005_S455NY0MB55506 diff --git a/hosts/taishan200-2280-2s48p-512g--a105 b/hosts/taishan200-2280-2s48p-512g--a105 index 66a0aca1f7b0..5a295e906de7 100644 --- a/hosts/taishan200-2280-2s48p-512g--a105 +++ b/hosts/taishan200-2280-2s48p-512g--a105 @@ -1,9 +1,10 @@ <<: taishan200-2280-2s48p-512g -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-350000399b891945d - /dev/disk/by-id/scsi-350000399b8919135 - /dev/disk/by-id/scsi-350000399b8919699 +rootfs_disk: - /dev/disk/by-id/scsi-350000399b89190d5 mac_addr: - 44:67:47:d7:76:c4 diff --git a/hosts/taishan200-2280-2s48p-512g--a95 b/hosts/taishan200-2280-2s48p-512g--a95 index 213d10f5ee77..ab649a3694c9 100644 --- a/hosts/taishan200-2280-2s48p-512g--a95 +++ b/hosts/taishan200-2280-2s48p-512g--a95 @@ -2,11 +2,12 @@ arch: aarch64 nr_node: 4 nr_cpu: 96 memory: 512G -nr_hdd_partitions: 4 +nr_hdd_partitions: 3 hdd_partitions: - /dev/disk/by-id/scsi-35000039988792e31 - /dev/disk/by-id/scsi-35000039988793705 - /dev/disk/by-id/scsi-35000039988792aa5 +rootfs_disk: - /dev/disk/by-id/scsi-350000399887932fd mac_addr: - cc:05:77:fd:fb:61 diff --git a/hosts/taishan200-2280-2s64p-256g--a110 b/hosts/taishan200-2280-2s64p-256g--a110 index 50625bc632ed..415baa86f1ff 100644 --- a/hosts/taishan200-2280-2s64p-256g--a110 +++ b/hosts/taishan200-2280-2s64p-256g--a110 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b6079 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b69bd mac_addr: - 44:67:47:e9:79:3d diff --git a/hosts/taishan200-2280-2s64p-256g--a112 b/hosts/taishan200-2280-2s64p-256g--a112 index 30c0f805c2c4..e53c76b77fe4 100644 --- a/hosts/taishan200-2280-2s64p-256g--a112 +++ b/hosts/taishan200-2280-2s64p-256g--a112 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399888173c1 +rootfs_disk: - /dev/disk/by-id/scsi-350000399888169c9 mac_addr: - 44:a1:91:9d:f3:3a diff --git a/hosts/taishan200-2280-2s64p-256g--a113 b/hosts/taishan200-2280-2s64p-256g--a113 index 826b18d6024d..015daaf0bcab 100644 --- a/hosts/taishan200-2280-2s64p-256g--a113 +++ b/hosts/taishan200-2280-2s64p-256g--a113 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000039988700409 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886bfb25 mac_addr: - 44:67:47:85:ed:b7 diff --git a/hosts/taishan200-2280-2s64p-256g--a114 b/hosts/taishan200-2280-2s64p-256g--a114 index f3a4d68c87c2..448b2e95b640 100644 --- a/hosts/taishan200-2280-2s64p-256g--a114 +++ b/hosts/taishan200-2280-2s64p-256g--a114 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399785bf105 +rootfs_disk: - /dev/disk/by-id/scsi-350000399785bf189 mac_addr: - 44:a1:91:9d:d5:44 diff --git a/hosts/taishan200-2280-2s64p-256g--a116 b/hosts/taishan200-2280-2s64p-256g--a116 index 5ca6cd4c7b2f..4b8dc936fe6d 100644 --- a/hosts/taishan200-2280-2s64p-256g--a116 +++ b/hosts/taishan200-2280-2s64p-256g--a116 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b6fad +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b6ba9 mac_addr: - 44:67:47:85:f0:af diff --git a/hosts/taishan200-2280-2s64p-256g--a117 b/hosts/taishan200-2280-2s64p-256g--a117 index 0f104951e8fd..696943cef060 100644 --- a/hosts/taishan200-2280-2s64p-256g--a117 +++ b/hosts/taishan200-2280-2s64p-256g--a117 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b57d1 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b720d mac_addr: - 44:67:47:85:f4:0b diff --git a/hosts/taishan200-2280-2s64p-256g--a118 b/hosts/taishan200-2280-2s64p-256g--a118 index 9a93e034527e..4bcfeea10d66 100644 --- a/hosts/taishan200-2280-2s64p-256g--a118 +++ b/hosts/taishan200-2280-2s64p-256g--a118 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-35000039988707e91 +rootfs_disk: - /dev/disk/by-id/scsi-350000399887080e5 mac_addr: - 44:67:47:e9:78:cf diff --git a/hosts/taishan200-2280-2s64p-256g--a119 b/hosts/taishan200-2280-2s64p-256g--a119 index 57da86389ada..92d341a5a487 100644 --- a/hosts/taishan200-2280-2s64p-256g--a119 +++ b/hosts/taishan200-2280-2s64p-256g--a119 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-350000399886b6be9 +rootfs_disk: - /dev/disk/by-id/scsi-350000399886b5dbd mac_addr: - 44:67:47:e9:79:65 diff --git a/hosts/taishan200-2280-2s64p-256g--a121 b/hosts/taishan200-2280-2s64p-256g--a121 index 9c87315d14c0..94d466828a85 100644 --- a/hosts/taishan200-2280-2s64p-256g--a121 +++ b/hosts/taishan200-2280-2s64p-256g--a121 @@ -1,6 +1,6 @@ memory: 256G -nr_ssd_partitions: 1 -ssd_partitions: +nr_ssd_partitions: 0 +rootfs_disk: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43943 mac_addr: - 44:67:47:e9:76:95 diff --git a/hosts/taishan200-2280-2s64p-256g--a123 b/hosts/taishan200-2280-2s64p-256g--a123 index f7650926e30e..cfc9c58df7b6 100644 --- a/hosts/taishan200-2280-2s64p-256g--a123 +++ b/hosts/taishan200-2280-2s64p-256g--a123 @@ -1,7 +1,8 @@ memory: 128G -nr_hdd_partitions: 2 +nr_hdd_partitions: 1 hdd_partitions: - /dev/disk/by-id/scsi-3500003998881eb19 +rootfs_disk: - /dev/disk/by-id/scsi-3500003998881e9a1 mac_addr: - 44:a1:91:9d:f5:4a diff --git a/hosts/taishan200-2280-2s64p-256g--a124 b/hosts/taishan200-2280-2s64p-256g--a124 index 7e7bb4222377..975dab549b2b 100644 --- a/hosts/taishan200-2280-2s64p-256g--a124 +++ b/hosts/taishan200-2280-2s64p-256g--a124 @@ -1,6 +1,6 @@ memory: 256G -nr_ssd_partitions: 1 -ssd_partitions: +nr_ssd_partitions: 0 +rootfs_disk: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB44344 mac_addr: - 44:67:47:e9:76:9f diff --git a/hosts/taishan200-2280-2s64p-256g--a5 b/hosts/taishan200-2280-2s64p-256g--a5 index e6c11b3def19..37881cce5b20 100644 --- a/hosts/taishan200-2280-2s64p-256g--a5 +++ b/hosts/taishan200-2280-2s64p-256g--a5 @@ -1,7 +1,7 @@ memory: 256G -nr_hdd_partitions: 1 +nr_hdd_partitions: 0 nr_ssd_partitions: 1 -hdd_partitions: +rootfs_disk: - /dev/disk/by-id/scsi-350000399981306c5 ssd_partitions: - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000328 diff --git a/hosts/taishan200-2280-2s64p-256g--a6 b/hosts/taishan200-2280-2s64p-256g--a6 index 09441939752c..dc11bb047fc9 100644 --- a/hosts/taishan200-2280-2s64p-256g--a6 +++ b/hosts/taishan200-2280-2s64p-256g--a6 @@ -1,7 +1,7 @@ memory: 256G -nr_hdd_partitions: 1 +nr_hdd_partitions: 0 nr_ssd_partitions: 1 -hdd_partitions: +rootfs_disk: - /dev/disk/by-id/scsi-350000399981344d9 ssd_partitions: - /dev/disk/by-id/nvme-HWE36P43012M000N_032BEJ10GC000301 diff --git a/hosts/taishan200-2280-2s64p-256g--a7 b/hosts/taishan200-2280-2s64p-256g--a7 index 07701d3cf3f9..4329ebfaff1e 100644 --- a/hosts/taishan200-2280-2s64p-256g--a7 +++ b/hosts/taishan200-2280-2s64p-256g--a7 @@ -1,7 +1,7 @@ memory: 256G -nr_hdd_partitions: 1 +nr_hdd_partitions: 0 nr_ssd_partitions: 1 -hdd_partitions: +rootfs_disk: - /dev/disk/by-id/scsi-35000039998134611 ssd_partitions: - /dev/disk/by-id/ata-SAMSUNG_MZ7LH480HAHQ-00005_S45PNA2MB43104 -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/kibana: upgrade and addition
by Wu Zhende 31 Mar '21

31 Mar '21

1. upgrade from 7.6.2 to 7.11.1 2. add a new kibana for data. one display log and one display data 3. delete logtrail plugin because it don't support in new kibana version Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana-logging/Dockerfile | 12 ++++++++ container/kibana-logging/build | 16 ++++++++++ container/kibana-logging/start | 31 +++++++++++++++++++ container/kibana-logging/start-depends | 1 + container/kibana/Dockerfile | 10 ------- container/kibana/build | 8 ++--- container/kibana/logtrail.json | 41 -------------------------- container/kibana/start | 14 ++++----- container/kibana/start-depends | 2 +- 9 files changed, 72 insertions(+), 63 deletions(-) create mode 100644 container/kibana-logging/Dockerfile create mode 100755 container/kibana-logging/build create mode 100755 container/kibana-logging/start create mode 100755 container/kibana-logging/start-depends delete mode 100644 container/kibana/logtrail.json diff --git a/container/kibana-logging/Dockerfile b/container/kibana-logging/Dockerfile new file mode 100644 index 0000000..970eb5a --- /dev/null +++ b/container/kibana-logging/Dockerfile @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +ARG BASE_IMAGE + +FROM ${BASE_IMAGE} + +# docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 + +MAINTAINER Wu Zhende <wuzhende666(a)163.com> + +RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml diff --git a/container/kibana-logging/build b/container/kibana-logging/build new file mode 100755 index 0000000..9c75aeb --- /dev/null +++ b/container/kibana-logging/build @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative '../defconfig' + +docker_skip_rebuild "kibana:7.11.1" + +BASE_IMAGE_DICT = { + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana-logging/start b/container/kibana-logging/start new file mode 100755 index 0000000..f2169a6 --- /dev/null +++ b/container/kibana-logging/start @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + LOGGING_ES_HOST + LOGGING_ES_PORT +] + +defaults = relevant_defaults(names) +LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' +LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' + +docker_rm 'kibana-logging' + +cmd = %W[ + docker run + --restart=always + --name kibana-logging + -v /etc/localtime:/etc/localtime:ro + -d + -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} + -p 20000:5601 + kibana:7.11.1 +] + +system(*cmd) diff --git a/container/kibana-logging/start-depends b/container/kibana-logging/start-depends new file mode 100755 index 0000000..66c1996 --- /dev/null +++ b/container/kibana-logging/start-depends @@ -0,0 +1 @@ +logging-es diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 60b889d..970eb5a 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -10,13 +10,3 @@ FROM ${BASE_IMAGE} MAINTAINER Wu Zhende <wuzhende666(a)163.com> RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml - -USER root -RUN yum -y install wget \ - && wget https://github.com/sivasamyk/logtrail/releases/download/v0.1.31/logtrail-7.… -O /logtrail-7.6.2-0.1.31.zip - -USER 1090 -RUN ./bin/kibana-plugin install file:///logtrail-7.6.2-0.1.31.zip \ - && rm -rf /tmp/logtrail-7.6.2-0.1.31.zip - -COPY --chown=1090:1090 logtrail.json /usr/share/kibana/plugins/logtrail/logtrail.json diff --git a/container/kibana/build b/container/kibana/build index 9a1fb6d..9c75aeb 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -5,12 +5,12 @@ require_relative '../defconfig' -docker_skip_rebuild "kibana:7.6.2" +docker_skip_rebuild "kibana:7.11.1" BASE_IMAGE_DICT = { - 'aarch64' => 'gagara/kibana-oss-arm64:7.6.2', - 'x86_64' => 'kibana:7.6.2' }.freeze + 'aarch64' => 'jamesgarside/kibana:7.11.1', + 'x86_64' => 'kibana:7.11.1' }.freeze BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -system "docker build -t kibana:7.6.2 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." diff --git a/container/kibana/logtrail.json b/container/kibana/logtrail.json deleted file mode 100644 index c63f2ac..0000000 --- a/container/kibana/logtrail.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "version": 2, - "index_patterns": [{ - "es": { - "default_index": "*", - "allow_url_parameter": false, - "timezone": "CST" - }, - "tail_interval_in_seconds": 10, - "es_index_time_offset_in_seconds": 0, - "display_timezone": "CST", - "display_timestamp_format": "YYYY MMM DD HH:mm:ss", - "max_buckets": 500, - "nested_objects": false, - "default_time_range_in_days": 30, - "max_hosts": 100, - "max_events_to_keep_in_viewer": 10000, - "default_search": "", - "fields": { - "mapping": { - "timestamp": "time", - "display_timestamp" : "time", - "hostname": "container_name", - "program": "tags", - "message": "log" - }, - "message_format": "{{{log}}}", - "keyword_suffix": "keyword" - }, - "color_mapping": { - "field": "level", - "mapping": { - "ERROR": "#FF0000", - "WARN": "#FFEF96", - "DEBUG": "#B5E7A0", - "TRACE": "#CFE0E8", - "INFO": "#339999" - } - } - }] -} diff --git a/container/kibana/start b/container/kibana/start index 26a4944..ac74ecc 100755 --- a/container/kibana/start +++ b/container/kibana/start @@ -7,13 +7,13 @@ require 'set' require_relative '../defconfig.rb' names = Set.new %w[ - LOGGING_ES_HOST - LOGGING_ES_PORT + ES_HOST + ES_PORT ] defaults = relevant_defaults(names) -LOGGING_ES_HOST = defaults['LOGGING_ES_HOST'] || '172.17.0.1' -LOGGING_ES_PORT = defaults['LOGGING_ES_PORT'] || '9202' +ES_HOST = defaults['ES_HOST'] || '172.17.0.1' +ES_PORT = defaults['ES_PORT'] || '9200' docker_rm 'kibana' @@ -23,9 +23,9 @@ cmd = %W[ --name kibana -v /etc/localtime:/etc/localtime:ro -d - -e ELASTICSEARCH_HOSTS=http://#{LOGGING_ES_HOST}:#{LOGGING_ES_PORT} - -p 20000:5601 - kibana:7.6.2 + -e ELASTICSEARCH_HOSTS=http://#{ES_HOST}:#{ES_PORT} + -p 20014:5601 + kibana:7.11.1 ] system(*cmd) diff --git a/container/kibana/start-depends b/container/kibana/start-depends index 66c1996..8357fca 100755 --- a/container/kibana/start-depends +++ b/container/kibana/start-depends @@ -1 +1 @@ -logging-es +es -- 2.23.0

1 0

[PATCH compass-ci] lib/stats_wrapper.rb: warn if stats-script doesn't exists
by Lu Weitao 31 Mar '21

31 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/stats_wrapper.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/stats_wrapper.rb b/lib/stats_wrapper.rb index faa81df..d7ef9ce 100644 --- a/lib/stats_wrapper.rb +++ b/lib/stats_wrapper.rb @@ -29,6 +29,11 @@ module StatsWrapper @stats_group = program_time || program @log = "#{RESULT_ROOT}/#{@stats_group}" + unless File.exist?("#{PROGRAM_DIR}/#{@program}") + warn "#{PROGRAM_DIR}/#{@program} doesn't exist" + return + end + return unless File.exist?("#{(a)log}.yaml") || pretreatment return unless create_tmpfile -- 2.23.0

1 0

Re: [PATCH v2 compass-ci] container/ssh-r/start: rename var for vm_name to docker_name
by Li Ping 31 Mar '21

31 Mar '21

rename var from vm_name to docker_name ---------^ Thanks, Ping Reply-To: In-Reply-To: <20210330074124.1780870-1-weijihuiall(a)163.com> On Tue, Mar 30, 2021 at 03:41:24PM +0800, Wei Jihui wrote: >vm_name --> docker_name > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > container/ssh-r/start | 6 +++--- > 1 file changed, 3 insertions(+), 3 deletions(-) > >diff --git a/container/ssh-r/start b/container/ssh-r/start >index 68a63f2..47a29eb 100755 >--- a/container/ssh-r/start >+++ b/container/ssh-r/start >@@ -4,14 +4,14 @@ > > . $CCI_SRC/container/defconfig.sh > >-vm_name=ssh_r >+docker_name=ssh_r > >-docker_rm ${vm_name} >+docker_rm ${docker_name} > > cmd=( > docker run > --restart=always >- --name ${vm_name} >+ --name ${docker_name} > -e GATEWAY_PORTS=true > -e TCP_FORWARDING=true > -d >-- >2.23.0 >

1 0

[PATCH compass-ci] providers/docker: replace URI.open to %x(curl url)
by Cao Xueliang 31 Mar '21

31 Mar '21

curl can suspend when the response need more time. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- providers/docker/docker.rb | 9 ++------- 1 file changed, 2 insertions(+), 7 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 51017f3..db0ccee 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -39,15 +39,10 @@ def del_host2queues(hostname) end def parse_response(url) - response = nil - URI.open(url) do |http| - response = http.read - end + response = %x(curl #{url}) hash = response.is_a?(String) ? JSON.parse(response) : nil if hash.nil? || !hash.key?('job') - puts '..........' - puts 'no job now' - puts '..........' + puts response return nil end return hash -- 2.23.0

1 0

[PATCH compass-ci] docker/run.sh: do not forward logs
by Wu Zhende 31 Mar '21

31 Mar '21

After the configuration, docker logs are dumped tp fluentd. The temporary docker also generates an index in logging-es. This is unnecessary and wastes es resources. Save temporary docker logs to the local host. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/run.sh | 2 ++ 1 file changed, 2 insertions(+) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 80950dd..516ebbc 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -36,6 +36,8 @@ cmd=( -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro -v ${busybox_path}:/usr/local/bin/busybox + --log-driver json-file + --log-opt max-size=10m --oom-score-adj="-1000" ${docker_image} /root/sbin/entrypoint.sh -- 2.23.0

1 0

[PATCH lkp-tests] stats/ansible_test:
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] optimize the output of error_id standard format: @module_name:error_id before: error.[message].Could-not-find-or-access-RedHat-20-yml-Searched-in-root-ansible-roles-ansible-role-postgresql-vars-RedHat-20-yml-root-ansible- roles-ansible-role-postgresql-RedHat-20-yml-root-ansible-roles-ansible-role-postgresql-tasks-vars-RedHat-20-yml-root-ansible-roles-ansible-role- postgresql-tasks-RedHat-20-yml-root-ansible-vars-RedHat-20-yml-root-ansible-RedHat-20-yml-on-the-Ansible-Controller-If-you-are-using-a-module- and-expect-the-file-to-exist-on-the-remote-see-the-remote_src-option: 1 after: @inculde_vars:Could-not-find-or-access-RedHat-20-yml: 1 Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 104 +++++++++++++++++++++++++++++++++++++-------- 1 file changed, 87 insertions(+), 17 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 3d3b0b0cb..0ede72c8f 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,40 +1,110 @@ #!/usr/bin/env ruby require 'json' +require 'yaml' -def output_error(error_msg, error_key) - return if error_msg.nil? - +def output_error(error_msg) if error_msg.is_a? Array error_msg.each do |i| error_id = common_error_id i - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end elsif error_msg.is_a? String error_id = common_error_id error_msg - puts "error.[#{error_key}].#{error_id}: 1" - puts "error.#{error_id}.message: #{@ansible_failed_info}" + puts "@#{@module_name['action']}:#{error_id}: 1" + puts "@#{@module_name['action']}:#{error_id}.message: #{@ansible_failed_info}" end end def common_error_id(line) - line.gsub!(/[^\w]/, '-') + line.gsub!(/[^\w]/, '--') + line.gsub!(/[0-9]{3,}/, '#') line.gsub!(/-+/, '-') # Replace multiple consecutive '-' with a single one line.gsub!(/^-|-$/, '') line end -while (line = STDIN.gets) - next unless line =~ /(FAILED!|failed:).*=>(.*)/ +def parse_msg(ansible_failed_json) + case ansible_failed_json['msg'] + when /package (.*64)/ + output_error "cannot install the #{$1} for the job" + when /(The error appears to be in .*')/ + output_error $1 + when /(Failed to import the required Python library .*python3\.)/ + output_error $1 + #when /(An unhandled exception occurred while running the lookup plugin .*'\.)/ + when /original message: (.*)/ + output_error $1 + when /(Unable to start service (\w{1,}):)/ + output_error $1 + when /(Failed to download metadata for repo .*'):/ + output_error $1 + when /(Failed to download packages: (.*):)/ + output_error $2 + when /(Failed to find required .* in paths):/ + output_error $1 + when /(Failure downloading .*),/ + output_error $1 + when /(Could not find or access.*)'/ + output_error $1 + when /(Unsupported parameters) for/ + output_error $1 + when /aajdhch/ + output_error $1 + when /non-zero return code|Failed to install some of the specified packages/ + return + else + output_error ansible_failed_json['msg'] + end +end - @ansible_failed_info = $2 - next if @ansible_failed_info.empty? +def parse_stderr(ansible_failed_json) + case ansible_failed_json['stderr'] + when /^$/ + return + else + ansible_failed_json['stderr_lines'].each do |i| + output_error i + end + end +end - ansible_failed_json = JSON.parse @ansible_failed_info +def parse_message(ansible_failed_json) + case ansible_failed_json['message'] + when /(Could not find or access.*)'/ + output_error $1 + else + output_error ansible_failed_json['message'] + end +end - output_error ansible_failed_json['msg'],'msg' - output_error ansible_failed_json['message'],'message' - output_error ansible_failed_json['cmd'],'cmd' - output_error ansible_failed_json['failures'],'failures' +def parse_failures(ansible_failed_json) + ansible_failed_json['failures'].each do |i| + case ansible_failed_json['failures'] + when /.*/ + p '1' + else + output_error ansible_failed_json['failures'] + end + end +end + +while (line = STDIN.gets) + case line + when /({'action'.*})/ + @module_name = YAML.load($1) + puts "module_name: #{@module_name['action']}" + when /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + parse_msg ansible_failed_json unless ansible_failed_json['msg'].nil? + parse_stderr ansible_failed_json unless ansible_failed_json['stderr'].nil? + parse_message ansible_failed_json unless ansible_failed_json['message'].nil? + parse_failures ansible_failed_json unless ansible_failed_json['failures'].nil? + end end -- 2.23.0

1 1