December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/3] lib/json_logger: support env
by Wu Zhende 01 Dec '20

01 Dec '20

API can set env, after set env will add some env info to log. Add: status_code/method/resource/elapsed Like this: {"level_num":1,"level":"INFO","time":"2020-12-01 20:03:19+08:00","message":"test","status_code":200,"method":"POST","resource":"/submit_job","elapsed":"133.68µs"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/json_logger.cr | 65 ++++++++++++++++++++++++++++++------------ 1 file changed, 47 insertions(+), 18 deletions(-) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 371c042..2b5ef6a 100755 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -6,27 +6,56 @@ require "json" require "any_merge" class JSONLogger < Logger - FORMATTER = Logger::Formatter.new do | severity, datetime, progname, msg, io| - level_num = severity.to_i32 - logger_hash = JSON.parse(%({"level_num": #{level_num}, - "level": "#{severity}", - "time": "#{datetime}" - })).as_h - - logger_hash.any_merge!({"progname" => progname}) unless progname.empty? - logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 - - begin - message = JSON.parse(msg).as_h - rescue - message = {"message" => msg} + def initialize(logdev = STDOUT, formatter = my_formatter) + @env = nil + @env_info = Hash(String, String | Int32).new + super(logdev, formatter: formatter) + end + + def my_formatter + Logger::Formatter.new do | severity, datetime, progname, msg, io| + get_env_info(@env.as(HTTP::Server::Context)) if @env + level_num = severity.to_i32 + logger_hash = JSON.parse(%({"level_num": #{level_num}, + "level": "#{severity}", + "time": "#{datetime}" + })).as_h + + logger_hash.any_merge!({"progname" => progname}) unless progname.empty? + logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 + + begin + message = JSON.parse(msg).as_h + rescue + message = {"message" => msg} + end + logger_hash.any_merge!(message) + logger_hash.any_merge!(@env_info) + + io << logger_hash.to_json end - logger_hash.any_merge!(message) + end - io << logger_hash.to_json + private def get_env_info(env : HTTP::Server::Context) + @env_info["status_code"] = env.response.status_code + @env_info["method"] = env.request.method + @env_info["resource"] = env.request.resource + + elapsed = get_elapsed(env) + @env_info["elapsed"] = elapsed.to_s if elapsed end - def initialize(logdev = STDOUT, formatter = FORMATTER) - super(logdev, formatter: formatter) + private def get_elapsed(env : HTTP::Server::Context) + start_time = env.get?("start_time") + return unless start_time + + elapsed = (Time.monotonic - start_time.as(Time::Span)).total_milliseconds + return "#{elapsed.round(2)}ms" if elapsed >= 1 + + "#{(elapsed * 1000).round(2)}µs" + end + + def set_env(env : HTTP::Server::Context) + @env = env end end -- 2.23.0

1 0

[PATCH v1 compass-ci 4/4] container: add locate_files in assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

An interface is provided to support obtaining the real path of the /srv/initrd/deps and pkg program. Example: Input: curl -H 'Content-Type: Application/json' -XPOST '172.17.0.1:8101/locate_files' -d '{"deps_files": ["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf.cgz"], "pkg_files": ["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/latest.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/latest.cgz"]}' Output: {"deps_files":["/srv/initrd/deps/nfs/openeuler/aarch64/20.03/perf/perf_20201104.cgz"], "pkg_files":["/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/plzip/1-4.cgz", "/srv/initrd/pkg/nfs/openeuler/aarch64/20.03/stream/1-1.cgz"]} Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 31 +++++++++++++++++++++++ container/assistant/views/locate_files.rb | 21 +++++++++++++++ 2 files changed, 52 insertions(+) create mode 100755 container/assistant/routes.rb create mode 100644 container/assistant/views/locate_files.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb new file mode 100755 index 0000000..54d3670 --- /dev/null +++ b/container/assistant/routes.rb @@ -0,0 +1,31 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'sinatra' +require 'json' +require 'open3' + +require_relative './views/locate_files' + +set :bind, '0.0.0.0' +set :port, 8101 + +post '/locate_files' do + request.body.rewind + + begin + data = JSON.parse request.body.read + rescue JSON::ParserError + return [400, 'parse json params error'] + end + + begin + result = locate_files(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + [200, result.to_json] +end diff --git a/container/assistant/views/locate_files.rb b/container/assistant/views/locate_files.rb new file mode 100644 index 0000000..5769177 --- /dev/null +++ b/container/assistant/views/locate_files.rb @@ -0,0 +1,21 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +def locate_files(data) + result = {} + + data.each do |key, value| + temp = [] + value.each do |val| + val = val.strip + if File.exist?(val) + temp << File.realpath(val) + end + end + + result.merge!({ "#{key}": temp }) + end + + return result +end -- 2.23.0

2 2

[PATCH v1 compass-ci 3/4] container: start script for assistant container
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/start | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100755 container/assistant/start diff --git a/container/assistant/start b/container/assistant/start new file mode 100755 index 0000000..5f7822e --- /dev/null +++ b/container/assistant/start @@ -0,0 +1,21 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +docker_rm assistant + +cmd=( + docker run + --restart=always + --name assistant + -u nobody + -d + -p 8101:8101 + -v /srv/initrd:/srv/initrd + -v /etc/localtime:/etc/localtime:ro + debian:assistant +) + +"${cmd[@]}" -- 2.23.0

2 2

[PATCH v1 compass-ci 1/4] container: assistant container Dockerfile
by Cao Xueliang 01 Dec '20

01 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/Dockerfile | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 container/assistant/Dockerfile diff --git a/container/assistant/Dockerfile b/container/assistant/Dockerfile new file mode 100644 index 0000000..0287747 --- /dev/null +++ b/container/assistant/Dockerfile @@ -0,0 +1,14 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM debian + +ENV DEBIAN_FRONTEND noninteractive + +RUN apt-get update && \ + apt-get install -y ruby-json ruby-sinatra curl + +COPY routes.rb /usr/local/bin/ +COPY views /usr/local/bin/views/ + +CMD ["/usr/local/bin/routes.rb"] -- 2.23.0

2 2

[PATCH v2 compass-ci 4/4] container/defconfig: fix rubocop warning
by Liu Yinsi 01 Dec '20

01 Dec '20

defconfig.rb:23:12: C: [Corrected] Style/BlockDelimiters: Avoid using {...} for multi-line blocks. hash.map { |k, v| Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 026a183..320fba4 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -20,9 +20,9 @@ end def set_local_env hash = cci_defaults - hash.map { |k, v| + hash.map do |k, v| system "export #{k}=#{v}" - } + end end def docker_env(hash) -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/2] sbin: load PKGBUILD.src content
by Liu Shaofei 01 Dec '20

01 Dec '20

The contents of PKGBUILD.src is the following example: source=( "http://172.168.131.113:8800/initrd/build-tar/ansible/ansible-2.9.4.tar.gz" ) Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index 6f8b95af..498b4872 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -895,6 +895,7 @@ update_pkgver() { sed --follow-symlinks -i "s:^pkgrel=[^ ]*:pkgrel=1:" "$BUILDFILE" source_safe "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_safe "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_safe "PKGBUILD.src" local fullver=$(get_full_version) msg "$(gettext "Updated version: %s")" "$pkgbase $fullver" else @@ -3699,6 +3700,7 @@ else fi source_buildfile "$BUILDFILE" [[ -n $PKGBUILD_TAG ]] && source_buildfile "$BUILDFILE-$PKGBUILD_TAG" + [[ -f PKGBUILD.src ]] && source_buildfile PKGBUILD.src fi # set defaults if they weren't specified in buildfile -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] tests: replace *.tar.* address with local address
by Liu Shaofei 01 Dec '20

01 Dec '20

When the address of source field is the format of "*.tar.*", we need to replace it with local address. The following example: source=("https://github.com/azukiapp/${pkgname}/archive/v${pkgver}.tar.gz") => source=("http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/v${pkgver}.tar.gz") Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..617dc0b7 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -98,6 +98,26 @@ replace_source() sed -i "s|^source=[^)]*|$1|g" "$2" } +# replace *.tar.* address from source field in PKGBUILD +replace_tar_source() +{ + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar\..+|.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue + + url="\"${local_url}\"" + } + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src +} + build_source_pkg() { local repo_dir="" @@ -116,6 +136,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } + replace_tar_source + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -- 2.23.0

1 0

[PATCH lab-z9] lab-z9: add cluster from a103 to a107
by Wei Jihui 01 Dec '20

01 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- cluster/cs-s1-a105-c4 | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 cluster/cs-s1-a105-c4 diff --git a/cluster/cs-s1-a105-c4 b/cluster/cs-s1-a105-c4 new file mode 100644 index 0000000..fa29b74 --- /dev/null +++ b/cluster/cs-s1-a105-c4 @@ -0,0 +1,22 @@ +switch: Switch-P10 +ip0: 2 +nodes: + taishan200-2280-2s48p-512g--a105: + roles: [ server ] + macs: [ "44:67:47:d7:76:c4" ] + + taishan200-2280-2s48p-256g--a103: + roles: [ client1 ] + macs: [ "44:67:47:c9:d8:b8" ] + + taishan200-2280-2s48p-256g--a104: + roles: [ client2 ] + macs: [ "44:67:47:d7:6d:f0" ] + + taishan200-2280-2s48p-256g--a106: + roles: [ client3 ] + macs: [ "44:67:47:86:1c:30" ] + + taishan200-2280-2s48p-256g--a107: + roles: [ client4 ] + macs: [ "84:46:fe:73:a2:71" ] -- 2.23.0

1 0

[PATCH v5 compass-ci] doc/manual: add how-to-log-in-the-machine.md
by Zhang Yu 01 Dec '20

01 Dec '20

Add a document for introducing how to log in the machine Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/manual/how-to-log-in-the-machine.md | 86 +++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 doc/manual/how-to-log-in-the-machine.md diff --git a/doc/manual/how-to-log-in-the-machine.md b/doc/manual/how-to-log-in-the-machine.md new file mode 100644 index 0000000..fefd718 --- /dev/null +++ b/doc/manual/how-to-log-in-the-machine.md @@ -0,0 +1,86 @@ +这篇文档将告诉你如何登陆测试环境 + +# 1. 前提条件 +请先学习: +* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 +* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 +* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 + +# 2. 操作方法 +## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： + +```shell + suite: host-info + category: functional + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + sshd: + # sleep at the bottom + sleep: 1h + + host-info: +``` + +## 2.2 以下有2种方式可以登录到测试机： +### 第一种：使用submit -m -c的方式： + 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 + 命令：submit -m -c host-info.yaml + 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： + 效果如下： + +```shell + hi6325@account-vm ~% submit -m -c atomic.yaml + submit atomic.yaml, got job_id=crystal.146528 + query=>{"job_id":["crystal.146528"]} + connect to ws://localhost:11310/filter + {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" + {"job_id": "crystal.146528", "job_state": "boot"} + {"job_id": "crystal.146528", "job_state": "download"} + "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} + {"job_state":"running","job_id":"crystal.146528"} + {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} + + root@vm-2p8g ~# +``` + +### 第二种：根据邮件信息使用ssh方式登录测试机： + 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 + 命令： submit host-info.yaml + 任务执行完成后，系统发送邮件内容如下： + +```shell + Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use + + Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, vm-2p8g-294828 has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + Regards + Compass-Ci +``` + + 可通过ssh方式登陆到测试环境： + 命令： ssh root(a)api.compass-ci.openeuler.org -p $port + 效果如下： + +```shell + hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 + + root@vm-2p8g ~# +``` -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/compare_matrixes.rb: change stddev to stddev_percent
by Lu Kaiyi 01 Dec '20

01 Dec '20

":stddev" doesn't exists in metric_value, change it to ":stddev_percent" to normally access the value of "standard deviation". Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_matrixes.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/compare_matrixes.rb b/lib/compare_matrixes.rb index 7a9870a..2bb6a61 100644 --- a/lib/compare_matrixes.rb +++ b/lib/compare_matrixes.rb @@ -460,7 +460,7 @@ def assign_metric_values(metrics_values, dim, metric, values) metrics_values[metric]['standard_deviation'] ||= {} metric_value = get_values(values, true) metrics_values[metric]['average'][dim] = metric_value[:average] - metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev] || 0 + metrics_values[metric]['standard_deviation'][dim] = metric_value[:stddev_percent] || 0 end def assign_metric_change(metrics_values) -- 2.23.0

1 0