- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/compare_data_format.rb: keep the size of x_params and dimension_values consistent
by Lu Kaiyi 02 Dec '20

02 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/compare_data_format.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/compare_data_format.rb b/lib/compare_data_format.rb index 484b186..1959045 100644 --- a/lib/compare_data_format.rb +++ b/lib/compare_data_format.rb @@ -49,12 +49,14 @@ class FormatEchartData def assign_echart_source(metric, value_type, dimensions) source = [] - source << @x_params + source << @x_params.clone dimensions.each do |dimension| dimension_values = [dimension] @x_params.each do |x_param| if @metrics_compare_results[x_param][metric] dimension_values << @metrics_compare_results[x_param][metric][value_type][dimension] + else + source[0].delete(x_param) end end source << dimension_values -- 2.23.0

1 0

[PATCH v3 compass-ci] doc/manual: 修改名字为如何登录测试机调测任务.md并增加和修改内容
by Zhang Yale 02 Dec '20

02 Dec '20

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- doc/manual/how-to-log-in-the-machine.md | 86 ------------------- ...03\346\265\213\344\273\273\345\212\241.md" | 72 ++++++++++++++++ 2 files changed, 72 insertions(+), 86 deletions(-) delete mode 100644 doc/manual/how-to-log-in-the-machine.md create mode 100644 "doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" diff --git a/doc/manual/how-to-log-in-the-machine.md b/doc/manual/how-to-log-in-the-machine.md deleted file mode 100644 index fefd718..0000000 --- a/doc/manual/how-to-log-in-the-machine.md +++ /dev/null @@ -1,86 +0,0 @@ -这篇文档将告诉你如何登陆测试环境 - -# 1. 前提条件 -请先学习: -* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 -* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 -* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 - -# 2. 操作方法 -## 2.1 在job.yaml里加上sshd和sleep字段, 以host-info.yaml任务为例： - -```shell - suite: host-info - category: functional - pub_key: <%= - begin - File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp - rescue - nil - end - %> - sshd: - # sleep at the bottom - sleep: 1h - - host-info: -``` - -## 2.2 以下有2种方式可以登录到测试机： -### 第一种：使用submit -m -c的方式： - 这种方式提交的任务会在指定的位置sleep并直接登陆到测试机中，适用于登陆测试环境后手动调试 - 命令：submit -m -c host-info.yaml - 任务提交完成后，当测试执行到sshd后会自动登陆到测试机器上： - 效果如下： - -```shell - hi6325@account-vm ~% submit -m -c atomic.yaml - submit atomic.yaml, got job_id=crystal.146528 - query=>{"job_id":["crystal.146528"]} - connect to ws://localhost:11310/filter - {"job_id": "crystal.146528", "result_root": "/srv/result/atomic/2020-12-01/vm-2p8g/openeuler-20.03-aarch64/1-1000/crystal.146528", "job_state": "set result root" - {"job_id": "crystal.146528", "job_state": "boot"} - {"job_id": "crystal.146528", "job_state": "download"} - "time":"2020-12-01 10:12:33","mac":"0a-2d-7b-d9-f8-b1","ip":"172.18.252.12","job_id":"crystal.146528","state":"running","testbox":"vm-2p8g.zhyl-453231"} - {"job_state":"running","job_id":"crystal.146528"} - {"job_id": "crystal.146528", "state": "set ssh port", "ssh_port": "51750", "tbox_name": "vm-2p8g.zhyl-453231"} - - root@vm-2p8g ~# -``` - -### 第二种：根据邮件信息使用ssh方式登录测试机： - 这种是使用submit方式提交的任务完成后，系统自动发送一封邮件提醒您可以在指定时间内登陆到测试环境 - 命令： submit host-info.yaml - 任务执行完成后，系统发送邮件内容如下： - -```shell - Subject: [NOTIFY Compass-ci] vm-2p8g-294828 ready to use - - Dear $my_username: - Thanks for your participation in software ecosystem! - According to your application, vm-2p8g-294828 has been provisioned. - The datails are as follows: - - Login: - ssh root(a)api.compass-ci.openeuler.org -p $port - Due time: - $deadline - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch - Regards - Compass-Ci -``` - - 可通过ssh方式登陆到测试环境： - 命令： ssh root(a)api.compass-ci.openeuler.org -p $port - 效果如下： - -```shell - hi6325@account-vm ~% ssh root(a)api.compass-ci.openeuler.org -p 51400 - - root@vm-2p8g ~# -``` diff --git "a/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" new file mode 100644 index 0000000..1b962fe --- /dev/null +++ "b/doc/manual/\345\246\202\344\275\225\347\231\273\345\275\225\346\265\213\350\257\225\346\234\272\350\260\203\346\265\213\344\273\273\345\212\241.md" @@ -0,0 +1,72 @@ +这篇文档将告诉你如何登陆测试环境去调测任务 + +# 1. 前提条件： +## 请先学习: +* [apply-account.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc…, 配置个人邮箱 +* [如何申请测试机.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua…, 并在本地生成RSA公私钥对 +* [submit命令详解.md](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma…, submit 命令详解 + +# 2. 操作方法 + +## 2.1 定义免密登录的yaml + +### 方法一：在测试机运行脚本之前进入测试机调测任务 + +在job.yaml里加上sshd和sleep字段，测试机在运行脚本之前sleep, 并免密登录进去，手动输入命令或脚本进行调试，以host-info.yaml为例： + +```yaml + suite: host-info + category: functional + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + sshd: + # sleep at the bottom + sleep: 1h + + host-info: +``` + +### 方法二：在测试任务运行失败时进入测试机调测任务 + +在job.yaml里加上pub_key和on_fail字段，并在on_fail下加上sshd和sleep字段，测试任务失败后，免密登录进去手动调试，以host-info.yaml为例： + +```yaml + suite: host-info + category: functional + + pub_key: <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + + host-info: + on_fail: + sshd: + sleep: 1000s +``` + +## 2.2 提交job.yaml + +### 方法一：直接提交 + + submit host-info.yaml + +在收到邮件后，按照邮件提示手动免密登录到执行机调测 + + ssh root(a)api.compass-ci.openeuler.org -p $port + +### 方法二：带参数提交 + + submit -m -c host-info.yaml + +该方法不用查看邮件，可自动免密登录到测试机 -- 2.23.0

1 0

[PATCH v2 compass-ci] params_group.rb: delete $user if tbox_group has
by Li Yuanchao 02 Dec '20

02 Dec '20

Some tbox_groups have $user in es. Such as vm-2p8g--$user or vm-2p8g.$user. But their config is the same, they should be compared together. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index cde40af..57cd5c1 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -67,10 +67,15 @@ def filter_groups(groups) end end +def get_tbox_group(tbox_group) + return tbox_group.gsub(/(--|\.).*$/, '') +end + def get_all_params(job) all_params = {} job.each_key do |param| all_params[param] = job[param] if COMMON_PARAMS.include?(param) + all_params[param] = get_tbox_group(job[param]) if param == 'tbox_group' next unless param == 'pp' pp_params = get_pp_params(job[param]) -- 2.23.0

2 3

[PATCH compass-ci 2/2] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..a897b57 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,31 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each_index do |index| + testboxes[index] = buckets[index]['key'] + end + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/web_backend: add a interface /get_testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

It would return a testbox list to web request Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index d68e9f2..11ba604 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -103,3 +103,10 @@ post '/compare_template' do request.body.rewind # in case someone already read it compare_template(request.body.read) end + +# GET /get_testboxes +# Response: +# - { total: total, testboxes[ 'testbox1', 'testbox2', ... ] } +get '/get_testboxes' do + query_testboxes +end -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 02 Dec '20

02 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..c788607 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,16 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic configurations, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: memory size of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query.rb: add traverse es data of function
by Cui Yili 02 Dec '20

02 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index c027aa9..4c99023 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -11,6 +11,7 @@ class ESQuery PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i def initialize(host = HOST, port = PORT, index: 'jobs') @index = index + @scroll_id = "" @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end @@ -33,6 +34,23 @@ class ESQuery @client.search index: 'jobs*', body: query end + def traverse_field(size) + if @scroll_id.empty? + query = { + query: { + bool: { + must: {match_all: {} } + } + }, size: size + } + result = @client.search index: @index, scroll: '10m', body: query + @scroll_id = result['_scroll_id'] + return result + else + @client.scroll scroll: '10m', scroll_id: @scroll_id + end + end + def query_by_id(id) @client.get_source({ index: @index, type: '_doc', id: id }) rescue Elasticsearch::Transport::Transport::Errors::NotFound -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] tests: replace *.tar.* address with local address
by Liu Shaofei 02 Dec '20

02 Dec '20

When the address of source field is the format of "*.tar.*", we need to replace it with local address. The following example: source=("https://github.com/azukiapp/${pkgname}/archive/v${pkgver}.tar.gz") => source=("http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/v${pkgver}.tar.gz") Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..cde2f0ed 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -98,6 +98,26 @@ replace_source() sed -i "s|^source=[^)]*|$1|g" "$2" } +# replace *.tar.* address from source field in PKGBUILD +replace_tar_source() +{ + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue + + url="\"${local_url}\"" + } + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src +} + build_source_pkg() { local repo_dir="" @@ -116,6 +136,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } + replace_tar_source + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -- 2.23.0

2 2

[PATCH compass-ci] params_group.rb: delete $user if tbox_group has
by Li Yuanchao 02 Dec '20

02 Dec '20

Some tbox_groups have $user in es. Such as vm-2p8g--$user or vm-2p8g.$user. But their config is the same, they should be compared together. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index cde40af..6d5c47b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -67,10 +67,17 @@ def filter_groups(groups) end end +def get_tbox_group(tbox_group) + tbox_group = tbox_group.split('--')[0] if tbox_group.include?('--') + tbox_group = tbox_group.split('.')[0] if tbox_group.include?('.') + return tbox_group +end + def get_all_params(job) all_params = {} job.each_key do |param| all_params[param] = job[param] if COMMON_PARAMS.include?(param) + all_params[param] = get_tbox_group(job[param]) if param == 'tbox_group' next unless param == 'pp' pp_params = get_pp_params(job[param]) -- 2.23.0

3 4

[PATCH compass-ci] sched: no need to check kernel config in initialized? function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7758e78..2a811cb 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,8 +146,8 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel() set_depends_initrd() + set_kernel() set_initrds_uri() set_lkp_server() set_sshr_info() @@ -345,7 +345,11 @@ class Job "SCHED_HOST", "SCHED_PORT"] - initialized_keys -= ["my_uuid"] + initialized_keys -= ["my_uuid", + "kernel_version", + "kernel_uri", + "kernel_params", + "linux_vmlinuz_path"] initialized_keys.each do |key| return false unless @hash.has_key?(key) -- 2.23.0

1 0