- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] fix: sched: add 2 fields into job: direct_macs and direct_ips
by Ren Wen 22 Oct '20

22 Oct '20

[Why] Nodes need to communicate with each other when testing cluster jobs. After adding 2 fileds: 'direct_macs' and 'direct_ips', can bind ips to macs. [HowTo] When you test a cluster job, need to config cluster file correctly. Below is the template: ```yaml <hostname>: roles: [ <"server">, ... ] macs: [ <mac>, ... ] ips: [ <ip>, ... ] <hostname>: roles: [ <"client">, ... ] macs: [ <mac>, ... ] ips: [ <ip>, ... ] ``` For 'ips': 1) should be the intranet IP 2) all ips in cluster file should be in the same network segment 3) subnet mask is 24. [InFuture] Set default ips when there's no ips in cluster file or only one host has ips. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 7215df5..a26091b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -219,7 +219,8 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + job["direct_macs"] = config["macs"].as_a.join(" ") + job["direct_ips"] = config["ips"].as_a.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

4 4

[PATCH lkp-tests] tests: fix lmbench3 tool adapt to aarch64 framwork
by Bai Jing 22 Oct '20

22 Oct '20

error: cat: results/x84_64-linux-gnu/taishan200-2280-2s48p-256g--a28.0 No such file or directory Signed-off-by: Bai Jing <799286817(a)qq.com> --- tests/lmbench3 | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/lmbench3 b/tests/lmbench3 index 8570a35a..f09c49e2 100755 --- a/tests/lmbench3 +++ b/tests/lmbench3 @@ -138,7 +138,7 @@ run_lmbench3_development() eval "$test=yes" - sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/x86_64-linux-gnu/lmbench + sed -i '/lat_pagefault -P $SYNC_MAX $FILE/i [ -f $FILE ] || dd if=/dev/zero of=$FILE count=1 bs=1G' bin/aarch64-linux-gnu/lmbench log_eval " ( echo $nr_threads @@ -177,7 +177,7 @@ run_lmbench3_development() show_result() { - cat results/x86_64-linux-gnu/$testbox.0 || { + cat results/aarch64-linux-gnu/$testbox.0 || { echo "no test result found" exit -1 } -- 2.23.0

3 3

[PATCH compass-ci] sbin: complier => compiler
by Cao Xueliang 22 Oct '20

22 Oct '20

[why] The container crystal-compiler build a image and named alpine:crystal-compiler, the sbin crystal-format use the image. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sbin/crystal-format | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/crystal-format b/sbin/crystal-format index 14f82b2..99f7a6a 100755 --- a/sbin/crystal-format +++ b/sbin/crystal-format @@ -15,7 +15,7 @@ cmd=( --rm -v $v_dir:$v_dir -w $PWD - alpine:crystal-complier + alpine:crystal-compiler crystal tool format "$@" ) -- 2.23.0

3 2

[PATCH compass-ci] auto_submit: testbox 2p8g => 2p16g for build-pkg
by Li Yuanchao 22 Oct '20

22 Oct '20

[why] 2p8g testbox only 1.3G left for build-pkg job, it's not enough. and there is error info write error: No space left on device Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 1096014..edfbfb1 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -4,4 +4,4 @@ compass-ci/compass-ci: - testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=centos os_version=7.6 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml archlinux: -- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib: fix btrfs-progs is not installed when os_mount=initramfs
by Hu Xuejiao 22 Oct '20

22 Oct '20

When job.yaml contains "fs: **", and /srv/initrd/deps/**/**/fs.cgz(btrfs-progs), it will execute mkfs.btrfs -f $rootfs_partition. but job.yaml don't container "fs: **" to appear warning: mkfs.btrfs command not found Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- lib/bootstrap.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index a7776d68..16789c5c 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -456,6 +456,7 @@ mount_rootfs() ROOTFS_DIR=/opt/rootfs mkdir -p $ROOTFS_DIR mount $rootfs_partition $ROOTFS_DIR || { + command -v mkfs.btrfs >/dev/null || return mkfs.btrfs -f $rootfs_partition mount $rootfs_partition $ROOTFS_DIR } -- 2.23.0

3 2

[PATCH lkp-tests 2/2] tests/iozone: add specified parameter for iozone tool
by Lu Kaiyi 22 Oct '20

22 Oct '20

[why] need to test write, rewrite, read, reread, random write, random read for 4g size and different block_size. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..f0c2878c 100755 --- a/tests/iozone +++ b/tests/iozone @@ -7,7 +7,7 @@ for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + cmd="iozone -r $block_size -s 4g -i 0 -i 1 -i 2 -f $mnt/iozonetest" log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

2 1

[PATCH lkp-tests 1/2] jobs/iozone.yaml: add specified block_size parameter
by Lu Kaiyi 22 Oct '20

22 Oct '20

[why] need to test different block_size value Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone.yaml | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/jobs/iozone.yaml b/jobs/iozone.yaml index f281186b..d17137e8 100644 --- a/jobs/iozone.yaml +++ b/jobs/iozone.yaml @@ -1,13 +1,17 @@ suite: iozone testcase: iozone category: benchmark +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m -disk: 2HDD +disk: 1HDD fs: -- xfs - ext4 -- btrfs iosched: - kyber -- 2.23.0

2 1

[PATCH lab-z9] hosts: add common configs
by Zhang Yu 22 Oct '20

22 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-512g | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-512g diff --git a/hosts/taishan200-2280-2s48p-512g b/hosts/taishan200-2280-2s48p-512g new file mode 100644 index 0000000..5a4126a --- /dev/null +++ b/hosts/taishan200-2280-2s48p-512g @@ -0,0 +1,4 @@ +nr_cpu: 96 +nr_node: 4 +memory: 512G +arch: aarch64 -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: put fork_stat to es
by Li Yuanchao 22 Oct '20

22 Oct '20

[why] We want to know whether git-mirror works well, so save last 10 times fetch_time and new_refs_time, and put them to es Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 37 +++++++++++++++++++++++++++++++------ 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 593d1eb..2b78523 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -98,8 +98,10 @@ class MirrorMain @fork_stat[stat_key] = { queued: false, priority: 0, - fetch_time: nil, - new_refs_time: nil + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 } end @@ -152,10 +154,9 @@ class MirrorMain feedback_info = @feedback_queue.pop(true) git_repo = feedback_info[:git_repo] - @fork_stat[git_repo][:queued] = false + update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - @fork_stat[git_repo][:priority] += 1 return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) @@ -196,8 +197,8 @@ class MirrorMain @git_info[git_repo] = YAML.safe_load(File.open(repodir)) @git_info[git_repo]['git_repo'] = git_repo @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] - es_repo_update(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) + es_repo_update(git_repo) @priority_queue.push git_repo, @priority @priority += 1 end @@ -255,11 +256,35 @@ class MirrorMain end end - def es_repo_update(git_repo, repo_info) + def es_repo_update(git_repo) + repo_info = @git_info[git_repo].merge(@fork_stat[git_repo]) + repo_info.delete(:cur_refs) if repo_info.key?(:cur_refs) body = { "doc": repo_info, "doc_as_upsert": true } @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) end + + def update_stat_fetch(git_repo) + @fork_stat[git_repo][:queued] = false + offset_fetch = @fork_stat[git_repo][:offset_fetch] + offset_fetch = 0 if offset_fetch >= 10 + @fork_stat[git_repo][:fetch_time][offset_fetch] = Time.now.to_s + @fork_stat[git_repo][:offset_fetch] = offset_fetch + 1 + end + + def update_stat_new_refs(git_repo) + @fork_stat[git_repo][:priority] += 1 + offset_new_refs = @fork_stat[git_repo][:offset_new_refs] + offset_new_refs = 0 if offset_new_refs >= 10 + @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s + @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 + end + + def update_fork_stat(git_repo, possible_new_refs) + update_stat_fetch(git_repo) + update_stat_new_refs(git_repo) if possible_new_refs + es_repo_update(git_repo) + end end -- 2.23.0

1 0

[PATCH v2 compass-ci] /src/lib/web_backend.rb: support multi-fields search in "/get_jobs"
by Zhang Yuhang 22 Oct '20

22 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 46 ++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..c3f6bef 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -248,23 +248,32 @@ def es_search(must, size, from) return es_query(query)['hits']['hits'], total end -def get_job(result) - job = {} - ALL_FIELDS.each do |f| - job[f] = result[f] +def get_jobs_result(result) + jobs = [] + result.each do |r| + job = {} + ALL_FIELDS.each do |f| + job[f] = r['_source'][f] + end + jobs << job end - job + jobs end -def search_job(git_repo, page_size, page_num) +def search_job(condition_fields, page_size, page_num) must = [] - must << { regexp: { upstream_repo: ".*#{git_repo}.*" } } if git_repo - jobs = [] - result, total = es_search(must, page_size, page_num * page_size) - result.each do |r| - jobs << get_job(r['_source']) + FIELDS.each do |field| + value = space_to_nil(condition_fields[field]) + next unless value + + must << if field.to_s == 'upstream_repo' + { regexp: { field => ".*#{value}.*" } } + else + { term: { field => value } } + end end - return jobs, total + result, total = es_search(must, page_size, page_num * page_size) + return get_jobs_result(result), total end def get_banner(git_repo, branches) @@ -291,17 +300,14 @@ def get_optimize_jobs_braches(jobs) end def get_jobs_body(params) - git_repo = space_to_nil(params[:upstream_repo]) - page_size = get_positive_number(params[:page_size], 20) - page_num = get_positive_number(params[:page_num], 1) - 1 - jobs, total = search_job(git_repo, page_size, page_num) + page_size = get_positive_number(params.delete(:page_size), 20) + page_num = get_positive_number(params.delete(:page_num), 1) - 1 + jobs, total = search_job(params, page_size, page_num) jobs, branches = get_optimize_jobs_braches(jobs) { total: total, - filter: { - upstream_repo: git_repo - }, - banner: get_banner(git_repo, branches), + filter: params, + banner: get_banner(params[:upstream_repo], branches), jobs: jobs, fields: FIELDS }.to_json -- 2.23.0

1 0