- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] container/rsync-server: modify data upload directory
by Cui Yili 29 Oct '20

29 Oct '20

[why] Modify data upload directory to upload qemu image. [how] Change the rsync.conf configuration and docker operating parameters. Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/rsync-server/rsyncd.conf | 6 ++++-- container/rsync-server/start | 4 ++-- 2 files changed, 6 insertions(+), 4 deletions(-) diff --git a/container/rsync-server/rsyncd.conf b/container/rsync-server/rsyncd.conf index 528e8bd..dd8f128 100644 --- a/container/rsync-server/rsyncd.conf +++ b/container/rsync-server/rsyncd.conf @@ -1,10 +1,12 @@ uid = lkp gid = lkp +port = 11387 use chroot = no +log file=/var/log/rsync.log -[result] -path = /result +[upload] +path = /srv/initrd/qemu-image read only = no max connections = 200 refuse options = delete diff --git a/container/rsync-server/start b/container/rsync-server/start index 4124192..6fa3de2 100755 --- a/container/rsync-server/start +++ b/container/rsync-server/start @@ -9,9 +9,9 @@ cmd=( docker run -id --name=rsync_server - -v /srv/result:/result + -v /srv/initrd/qemu-image:/srv/initrd/qemu-image -v /etc/localtime:/etc/localtime:ro - -p 873:873 + -p 11387:11387 rsync_server:latest ) -- 2.23.0

2 1

[PATCH compass-ci 1/2] auto group jobs with user-defined template
by Lu Weitao 29 Oct '20

29 Oct '20

auto group jobs with user-defined template to support template compare mode background: before compare, create job-groups for all jobs that query from ES with user's filter then, create matrices-groups according job-groups finally, compare each group matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 120 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 120 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index 0f07d26..4d10dbe 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -113,3 +113,123 @@ end def remove_singleton(groups) groups.delete_if { |_k, v| v.length < 2 } end + +# -------------------------------------------------------------------------------------------------- +# auto group job_list by user's template +# Exammple: +# Input: +# 1. jobs_list. +# 2. params from template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# Output: +# eg: +# { +# '4k|1G' => { +# 'openeuler 20.03' => [ +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +# {'stats' => {'fio.write_iops' => 289661.878453}}, +# ... +# ], +# 'contos 7.6' => [...] +# }, +# '16K|1G' => {...}, +# ... +# } +# + +def auto_group_by_template(jobs_list, group_params, dimensions, metrics) + job_list = extract_jobs_list(jobs_list) + get_group_by_template(job_list, group_params, dimensions, metrics) +end + +def get_group_by_template(job_list, group_params, dimensions, metrics) + groups = {} + job_list.each do |job| + next unless job['stats'] + + group_key = get_user_group_key(job, group_params) + dimension = get_user_dimension(job, dimensions) + next unless group_key && dimension + + new_job = get_new_job_by_metrics(job, metrics) + next if new_job.empty? + + groups[group_key] ||= {} + groups[group_key][dimension] ||= [] + groups[group_key][dimension] << new_job + end + groups +end + +# @group_params Array(String) +# eg: +# ['block_size', 'package_size'] +# return eg: +# '4K|1G' +def get_user_group_key(job, group_params) + group_key_list = [] + group_params.each do |param| + if job.key?(param) + group_key_list << job[param] + next + end + job.each_value do |v| + if v.is_a?(Hash) && v.key?(param) + group_key_list << v[param] + break + end + end + end + return nil if group_key_list.size < group_params.size + + group_key_list.join('|') +end + +# @dimension Array(Hash) +# eg: +# [ +# {os => openeuler, os_version => 20.03}, +# {os => centos, os_version => 7.6} +# ] +# return eg: +# 'openeuler 20.03' +def get_user_dimension(job, dimensions) + dimension_list = [] + + dimensions.each do |dim| + dim.each_key do |key| + if job[key] == dim[key] + dimension_list << job[key] + end + end + break unless dimension_list.empty? + end + return nil if dimension_list.size < dimensions.size + + dimension_list.join(' ') +end + +# @metrics Array(String) +# eg: +# ["fio.read_iops", "fio.write_iops"] +# return new_job +# eg: +# {'stats' => {'fio.write_iops' => 312821.002387, 'fio.read_iops' => 212821.2387}}, +def get_new_job_by_metrics(job, metrics) + new_job = {} + metrics.each do |metric| + if job['stats'].key?(metric) + new_job['stats'] ||= {} + new_job['stats'][metric] = job['stats'][metric] + end + end + new_job +end -- 2.23.0

4 6

[PATCH lab-z9] hosts: delete repet arch field
by Zhang Yu 29 Oct '20

29 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-256g | 1 - hosts/taishan200-2280-2s64p-256g | 1 - 2 files changed, 2 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g b/hosts/taishan200-2280-2s48p-256g index 642e8e1..b326025 100644 --- a/hosts/taishan200-2280-2s48p-256g +++ b/hosts/taishan200-2280-2s48p-256g @@ -2,4 +2,3 @@ arch: aarch64 nr_cpu: 96 nr_node: 4 memory: 256G -arch: aarch64 diff --git a/hosts/taishan200-2280-2s64p-256g b/hosts/taishan200-2280-2s64p-256g index 16349d9..7c7eb5c 100644 --- a/hosts/taishan200-2280-2s64p-256g +++ b/hosts/taishan200-2280-2s64p-256g @@ -2,4 +2,3 @@ arch: aarch64 nr_cpu: 128 nr_node: 4 memory: 256G -arch: aarch64 -- 2.23.0

2 3

[PATCH compass-ci 2/2] create matrices-groups according to group jobs
by Lu Weitao 29 Oct '20

29 Oct '20

create matrices-groups according to group jobs for support user-defined template compare mode background: before compare, create job-groups for all jobs that query from ES with user's filter then, create matrices-groups according job-groups finally, compare each group matrices Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 05f96dc..61517d9 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -120,3 +120,27 @@ def combine_group_query_data(query_data, dims) end groups end + +# input: +# 1. query results from es_query +# 2. params from template include: +# groups_params(x_params): +# eg: ['block_size', 'package_size'] +# dimensions: +# eg: [ +# {'os' => 'openeuler', 'os_version' => '20.03'}, +# {'os' => 'centos', 'os_version' => '7.6'} +# ] +# metrics: +# eg: ['fio.read_iops', 'fio_write_iops'] +# return: group_matrix of Hash(String, Hash(String, matrix)) +def combine_user_group_query_data(query_data, groups_params, dimensions, metrics) + job_list = query_data['hits']['hits'] + groups = auto_group_by_template(job_list, groups_params, dimensions, metrics) + groups.each do |group_key, value| + value.each do |dimension_key, jobs| + groups[group_key][dimension_key] = create_matrix(jobs) + end + end + groups +end -- 2.23.0

3 4

[PATCH compass-ci] container/netdata: use json-file save logs
by Wu Zhende 29 Oct '20

29 Oct '20

[Why] netdata is aperformance check tool and does not need to send logs to fluentd for analysis. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/netdata/start | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/netdata/start b/container/netdata/start index a3009a9..58fb675 100755 --- a/container/netdata/start +++ b/container/netdata/start @@ -17,6 +17,8 @@ cmd=( -v /var/run/docker.sock:/var/run/docker.sock:ro -v /srv/cache/netdata_cache:/var/cache/netdata -v /srv/cache/netdata_lib:/var/lib/netdata + --log-driver json-file + --log-opt max-size=1g --cap-add SYS_PTRACE --security-opt apparmor=unconfined netdata/netdatalkp -- 2.23.0

1 0

[PATCH lab-z9] hosts: Delete duplicate field
by Zhang Yu 29 Oct '20

29 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-256g | 1 - hosts/taishan200-2280-2s64p-256g | 1 - 2 files changed, 2 deletions(-) diff --git a/hosts/taishan200-2280-2s48p-256g b/hosts/taishan200-2280-2s48p-256g index 642e8e1..b326025 100644 --- a/hosts/taishan200-2280-2s48p-256g +++ b/hosts/taishan200-2280-2s48p-256g @@ -2,4 +2,3 @@ arch: aarch64 nr_cpu: 96 nr_node: 4 memory: 256G -arch: aarch64 diff --git a/hosts/taishan200-2280-2s64p-256g b/hosts/taishan200-2280-2s64p-256g index 16349d9..7c7eb5c 100644 --- a/hosts/taishan200-2280-2s64p-256g +++ b/hosts/taishan200-2280-2s64p-256g @@ -2,4 +2,3 @@ arch: aarch64 nr_cpu: 128 nr_node: 4 memory: 256G -arch: aarch64 -- 2.23.0

1 0

[PATCH lkp-tests] tests: add get_cpu function
by Zhang Yu 29 Oct '20

29 Oct '20

[why] Obtain more host information from 'lscpu' Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- tests/host-info | 21 ++++++++++++++------- 1 file changed, 14 insertions(+), 7 deletions(-) diff --git a/tests/host-info b/tests/host-info index 46f43f93..84f796dc 100755 --- a/tests/host-info +++ b/tests/host-info @@ -1,18 +1,25 @@ #!/bin/bash -get_node() +get_memory() { - echo "nr_node: $(grep -Eo 'Node [0-9]+,' /proc/zoneinfo | uniq | wc -l)" - echo "nr_cpu: $(nproc)" memory_total=$(cat /proc/meminfo |grep MemTotal | awk '{print $2}') memory_new=$(awk 'BEGIN{printf "%0.2f", '$memory_total'/1024/1024/16}') memory=$(echo `expr $(echo $memory_new | awk '{print int($0)+1}') \* 16`)G echo "memory: $memory" } -get_model_name() +get_cpu() { - echo "model_name: $(lscpu |grep "Model name" | awk '{print $3}')" + echo $1: $(lscpu | awk -F': *' '/'$2':/{print $2}') +} + +get_cpu_info() +{ + cpu_info=("arch Architecture" "nr_node NUMA\snode$s$" "nr_cpu ^CPU$s$" "model_name Model\sname") + for info in "${cpu_info[@]}" + do + get_cpu $info + done } get_nr_disk() @@ -101,11 +108,11 @@ get_ipmi_ip() main() { - get_node + get_memory get_nr_disk get_partitions get_mac - get_model_name + get_cpu_info get_ipmi_ip } -- 2.23.0

1 0

[PATCH v6 compass-ci] add queue field into job.yaml
by Xiao Shenwei 29 Oct '20

29 Oct '20

use queue field decide which queue to enter queue field come from user's specify(job.yaml) or dafault Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/job.cr | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 8f5dd36..7fbc414 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -73,6 +73,7 @@ class Job tbox_group testbox lab + queue initrd_pkg initrd_deps initrds_uri @@ -139,6 +140,7 @@ class Job set_kernel_params() set_lkp_server() set_sshr_port() + set_queue() end private def append_init_field @@ -199,6 +201,16 @@ class Job self["result_service"] = "raw_upload" end + private def set_queue + return if @hash.has_key?("queue") + + # set default value + self["queue"] = tbox_group + if tbox_group.to_s.starts_with?(/(vm|dc)-/) + self["queue"] = "#{tbox_group}.#{arch}" + end + end + # if not assign tbox_group, set it to a match result from testbox # ?if job special testbox, should we just set tbox_group=textbox private def update_tbox_group_from_testbox -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code: add centos in os file
by Zhang Yale 29 Oct '20

29 Oct '20

[Why] when run dev-env in centos system, error like this: ./dev-env:line 41: ./os/centos: No such file or directory. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- sparrow/3-code/os/centos | 6 ++++++ 1 file changed, 6 insertions(+) create mode 100644 sparrow/3-code/os/centos diff --git a/sparrow/3-code/os/centos b/sparrow/3-code/os/centos new file mode 100644 index 0000000..efb4fcb --- /dev/null +++ b/sparrow/3-code/os/centos @@ -0,0 +1,6 @@ +#!/bin/bash +cat >> /etc/default/docker <<EOF +DOCKER_OPTS="--dns 114.114.114.114" +EOF + +systemctl restart docker -- 2.23.0

4 6

[PATCH v2 compass-ci] LICENSE: add document license
by Bai Jing 29 Oct '20

29 Oct '20

Signed-off-by: Bai Jing <799286817(a)qq.com> --- LICENSES/CCBY-4.0 | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 LICENSES/CCBY-4.0 diff --git a/LICENSES/CCBY-4.0 b/LICENSES/CCBY-4.0 new file mode 100644 index 0000000..abaf8d0 --- /dev/null +++ b/LICENSES/CCBY-4.0 @@ -0,0 +1,4 @@ +Copyright (c) 2020 Huawei Technologies Co., Ltd. +Your copying, use, modification, and distribution of this document is subject to the Creative Commons Attribution-ShareAlike 4.0 International Public License ("CC BY-SA 4.0"). +For ease of understanding, you can visit https://creativecommons.org/licenses/by-sa/4.0/to learn about the outline of CC BY-SA 4.0(but not a substitute). +The complete agreement of CC BY-SA 4.0 can be obtained from the following website: https://creativecommons.org/licenses/by-sa/4.0/legalcode. -- 2.23.0

2 2