March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] test/ansible_test: modify ansible code to output the module name
by Wang Chenglong 02 Apr '21

02 Apr '21

[Why] modify ansible code to output the name of the module for tasks execution. [How] print("{'action':'%s'}" % task_result.__dict__['_task_fields']['action']) {'action':'package'} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/tests/ansible_test b/tests/ansible_test index 56926bbed..1c19c2b0f 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -58,7 +58,18 @@ ansible_playbook_run() EOF [ $? -eq 0 ] && { echo "Playbook has been generated!" - } + } + [ $(command -v ansible-playbook) ] || { + echo "ansible-playbook: command not found" + exit 1 + } + + sed -i "536a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py + + sed -i "550a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py + ansible-playbook "$ansible_path"/site.yml if [ $? -eq 0 ]; then -- 2.23.0

2 1

[PATCH v2 lkp-tests] stat: optimize to obtain error for build DockerFile
by Liu Shaofei 02 Apr '21

02 Apr '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- stats/openeuler_docker.rb | 101 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 101 insertions(+) create mode 100755 stats/openeuler_docker.rb diff --git a/stats/openeuler_docker.rb b/stats/openeuler_docker.rb new file mode 100755 index 000000000..706b5b249 --- /dev/null +++ b/stats/openeuler_docker.rb @@ -0,0 +1,101 @@ +#!/usr/bin/env ruby + +def pre_handel(result, file_path) + status = false + repo_set = Set[] + sys_set = Set[] + + File.readlines(file_path).each do |line| + case line.chomp! + # Error: Unable to find a match: docker-registry mock xx + when /Error: Unable to find a match: (.+)/ + $1.split.each do |repo| + repo_set << repo + end + + # RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs + # yum swap: error: unrecognized arguments: install systemd systemd-libs + when /yum swap: error: .*: install (.+)/ + $1.split.each do |sys| + sys_set << sys + end + + # curl: (22) The requested URL returned error: 404 Not Found + # error: skipping https://dl.fedoraproject.org/pub/epel/bash-latest-7.noarch.rpm - transfer failed + when /.*error: .* (https.*)/ + result['requested-URL-returned.error'] = [1] + result['requested-URL-returned.error.message'] = [line.to_s] + status = true + + # Error: Unknown repo: 'powertools' + when /Error: Unknown repo: (.+)/ + repo = $1.delete!("'") + result["unknown-repo.#{repo}"] = [1] + result["unknown-repo.#{repo}.message"] = [line.to_s] + status = true + + # Error: Module or Group 'convert' does not exist. + when /Error: Module or Group ('[^\s]+')/ + repo = $1.delete!("'") + result["error.not-exist-module-or-group.#{repo}"] = [1] + result["error.not-exist-module-or-group.#{repo}.message"] = [line.to_s] + status = true + # /bin/sh: passwd: command not found + when /\/bin\/sh: (.+): command not found/ + result["sh.command-not-found.#{$1}"] = [1] + result["sh.command-not-found.#{$1}.message"] = [line.to_s] + status = true + end + + repo_set.each do |repo| + result["yum.error.Unable-to-find-a-match.#{repo}"] = [1] + result["yum.error.Unable-to-find-a-match.#{repo}.message"] = ["Error: Unable to find a match #{repo}"] + status = true + end + + sys_set.each do |sys| + result["yum.swap.error.unrecognized-arguments-install.#{sys}"] = [1] + result["yum.swap.error.unrecognized-arguments.#{sys}.message"] = + ["yum swap: error: unrecognized arguments install #{sys}"] + status = true + end + end + status +end + +def handle_unknown_error(_result, file_path) + line_num = %x(cat #{file_path} | grep -n 'Step ' | tail -1 | awk -F: '{print $1}') + + index = 1 + message = '' + File.readlines(file_path).each do |line| + if index == Integer(line_num) + message += line + else + index += 1 + end + end + + message = $1 if message =~ %r(\u001b\[91m(.+)) + message +end + +def openeuler_docker(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + + log_lines.each do |line| + next unless line =~ %r(([^\s]+).(build|run)\.fail) + + key, value = line.split(':') + key.chomp! + result[key] << value.to_i + + file_path = "#{RESULT_ROOT}/#{$1}" # $1 named by docker-image name + next unless File.exist?(file_path) + next if pre_handel(result, file_path) + + result["#{key}.message"] << handle_unknown_error(result, file_path) + end + + result +end -- 2.23.0

3 3

[PATCH v3 compass-ci] container/ssh-r/start: rename variable name from vm_name to docker_name
by Wei Jihui 31 Mar '21

31 Mar '21

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- container/ssh-r/start | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/container/ssh-r/start b/container/ssh-r/start index 68a63f2..47a29eb 100755 --- a/container/ssh-r/start +++ b/container/ssh-r/start @@ -4,14 +4,14 @@ . $CCI_SRC/container/defconfig.sh -vm_name=ssh_r +docker_name=ssh_r -docker_rm ${vm_name} +docker_rm ${docker_name} cmd=( docker run --restart=always - --name ${vm_name} + --name ${docker_name} -e GATEWAY_PORTS=true -e TCP_FORWARDING=true -d -- 2.23.0

1 0

[PATCH lkp-tests] test/ansible_test: modify ansible code to output the module name
by Wang Chenglong 31 Mar '21

31 Mar '21

[Why] modify ansible code to output the name of the module for tasks execution. [How] print("{'action':'%s'}" % task_result.__dict__['_task_fields']['action']) {'action':'package'} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/ansible_test | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/tests/ansible_test b/tests/ansible_test index 56926bbed..8c3e68c59 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -58,7 +58,17 @@ ansible_playbook_run() EOF [ $? -eq 0 ] && { echo "Playbook has been generated!" - } + } + [ $(command -v ansible-playbook) ] || { + echo "ansible-playbook: command not found" + exit 1 + } + + sed -i "536a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + ansible-playbook "$ansible_path"/site.yml + + sed -i "550a\ print$\"{\'action\':\'%s\'}\" \% task_result.__dict__[\'_task_fields\'][\'action\']$" \ + /usr/local/lib/python3.7/site-packages/ansible/plugins/strategy/__init__.py ansible-playbook "$ansible_path"/site.yml if [ $? -eq 0 ]; then -- 2.23.0

1 1

[PATCH compass-ci] replace alpine mirrors
by Liu Yinsi 31 Mar '21

31 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/result-webdav/Dockerfile | 1 + container/srv-http/Dockerfile | 2 ++ container/upload-libvirt-xml/Dockerfile | 1 + 3 files changed, 4 insertions(+) diff --git a/container/result-webdav/Dockerfile b/container/result-webdav/Dockerfile index db48bcb..2edd2a7 100644 --- a/container/result-webdav/Dockerfile +++ b/container/result-webdav/Dockerfile @@ -5,6 +5,7 @@ FROM openresty/openresty:alpine +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add bash RUN mkdir -p /var/log/nginx/ RUN adduser lkp -u 1090 -g 1090 -D diff --git a/container/srv-http/Dockerfile b/container/srv-http/Dockerfile index b7d0677..4c010b9 100644 --- a/container/srv-http/Dockerfile +++ b/container/srv-http/Dockerfile @@ -18,6 +18,8 @@ FROM nginx:alpine ADD root / +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories + RUN apk add --no-cache php7 php7-fpm php7-session php7-json php7-exif php7-imagick php7-gd php7-fileinfo RUN sed -i '/\[global\]/a daemonize = no' /etc/php7/php-fpm.conf diff --git a/container/upload-libvirt-xml/Dockerfile b/container/upload-libvirt-xml/Dockerfile index db48bcb..2edd2a7 100644 --- a/container/upload-libvirt-xml/Dockerfile +++ b/container/upload-libvirt-xml/Dockerfile @@ -5,6 +5,7 @@ FROM openresty/openresty:alpine +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk add bash RUN mkdir -p /var/log/nginx/ RUN adduser lkp -u 1090 -g 1090 -D -- 2.23.0

1 0

[PATCH compass-ci] service/scheduler: add event time
by Wu Zhende 31 Mar '21

31 Mar '21

example: "submit_time": "2021-03-31T16:01:07+0800" "boot_time": "2021-03-31T16:01:33+0800" "close_time": "2021-03-31T16:02:47+0800" Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 5 +++++ src/scheduler/close_job.cr | 5 ++++- src/scheduler/find_job_boot.cr | 1 + 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 5767f61..2d15aae 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -170,6 +170,11 @@ class Job set_sshr_info() set_queue() set_subqueue() + set_time("submit_time") + end + + def set_time(key) + self[key] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") end private def set_docker_os diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 746f70a..a112a38 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -15,6 +15,9 @@ class Sched job["job_state"] = job_state if job_state job["job_state"] = "complete" if job["job_state"] == "boot" + job.set_time("close_time") + @env.set "close_time", job["close_time"] + response = @es.set_job_content(job) if response["_id"] == nil # es update fail, raise exception @@ -34,7 +37,7 @@ class Sched mq_msg = { "job_id" => @env.get?("job_id").to_s, "job_state" => "close", - "time" => get_time + "time" => @env.get?("close_time").to_s } spawn mq_publish_confirm(JOB_MQ, mq_msg.to_json) end diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 2596f83..6e8d5d1 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -92,6 +92,7 @@ class Sched if job job.update({"testbox" => testbox}) job.set_result_root + job.set_time("boot_time") @log.info(%({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state" : "set result root"})) set_id2job(job) -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] lib/git_mirror.rb: fix es update fail
by Li Yuanchao 31 Mar '21

31 Mar '21

As the es update to 7.11, some compatibility error occurs. [error msg] git-mirror: warning: 299 Elasticsearch-7.11.1-ff17057114c2199c9c1bbecc727003a907c0db7a "[types removal] Specifying types in doc ument update requests is deprecated, use the endpoint /{index}/_update/{id} instead." Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 7f36be3..1f4e5b5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -308,11 +308,8 @@ class MirrorMain def es_repo_update(git_repo) repo_info = { 'git_repo' => git_repo, 'url' => @git_info[git_repo]['url'] } repo_info = repo_info.merge(@fork_stat[git_repo]) - body = { - "doc": repo_info, - "doc_as_upsert": true - } - @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) + body = repo_info + @es_client.index(index: 'repo', type: '_doc', id: git_repo, body: body) end def update_stat_fetch(git_repo) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: fix no positive? method error
by Li Yuanchao 31 Mar '21

31 Mar '21

The result of query es changed, the result['hits']['total'] changes from a number to a hash, so the positive? method can not be used. Now use the count method which contained by es_client. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index c967c8b..7f36be3 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -414,9 +414,9 @@ class MirrorMain new_refs_count: {} } query = { query: { match: { _id: git_repo } } } - result = @es_client.search(index: 'repo', body: query)['hits'] - return fork_stat unless result['total'].positive? + return fork_stat unless @es_client.count(index: 'repo', body: query)['count'].positive? + result = @es_client.search(index: 'repo', body: query)['hits'] fork_stat.each_key do |key| fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end -- 2.23.0

1 0

[PATCH lkp-tests] add self-test.yaml
by Liu Yinsi 31 Mar '21

31 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- jobs/self-test.yaml | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 jobs/self-test.yaml diff --git a/jobs/self-test.yaml b/jobs/self-test.yaml new file mode 100644 index 000000000..ad2dcd2a2 --- /dev/null +++ b/jobs/self-test.yaml @@ -0,0 +1,6 @@ +suite: self-test +category: functional + +deploy-cci: +multi-qemu: +multi-docker: -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow: add 8-testbox/multi-docker
by Liu Yinsi 31 Mar '21

31 Mar '21

For user locally deploy compass-ci, execute multi-docker in the background, it is convenient for users to submit jobs. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/8-testbox/multi-docker | 5 +++++ sparrow/install-tiny | 1 + 2 files changed, 6 insertions(+) create mode 100755 sparrow/8-testbox/multi-docker diff --git a/sparrow/8-testbox/multi-docker b/sparrow/8-testbox/multi-docker new file mode 100755 index 0000000..25cb82d --- /dev/null +++ b/sparrow/8-testbox/multi-docker @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +nohup $CCI_SRC/providers/multi-docker -n dc-8g -c 10 -q dc-8g.$(arch) & diff --git a/sparrow/install-tiny b/sparrow/install-tiny index e48bd25..40358a4 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -23,3 +23,4 @@ cd $CCI_SRC/sparrow || exit 5-build/os-ready 5-build/register-account 7-systemd/systemd-setup +8-testbox/multi-docker -- 2.23.0

1 0