October 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] auto_submit: testbox 2p8g => 2p16g for build-pkg
by Li Yuanchao 22 Oct '20

22 Oct '20

[why] 2p8g testbox only 1.3G left for build-pkg job, it's not enough. and there is error info write error: No space left on device Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 1096014..edfbfb1 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -4,4 +4,4 @@ compass-ci/compass-ci: - testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=centos os_version=7.6 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml archlinux: -- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib: fix btrfs-progs is not installed when os_mount=initramfs
by Hu Xuejiao 22 Oct '20

22 Oct '20

When job.yaml contains "fs: **", and /srv/initrd/deps/**/**/fs.cgz(btrfs-progs), it will execute mkfs.btrfs -f $rootfs_partition. but job.yaml don't container "fs: **" to appear warning: mkfs.btrfs command not found Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- lib/bootstrap.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index a7776d68..16789c5c 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -456,6 +456,7 @@ mount_rootfs() ROOTFS_DIR=/opt/rootfs mkdir -p $ROOTFS_DIR mount $rootfs_partition $ROOTFS_DIR || { + command -v mkfs.btrfs >/dev/null || return mkfs.btrfs -f $rootfs_partition mount $rootfs_partition $ROOTFS_DIR } -- 2.23.0

3 2

[PATCH lkp-tests 2/2] tests/iozone: add specified parameter for iozone tool
by Lu Kaiyi 22 Oct '20

22 Oct '20

[why] need to test write, rewrite, read, reread, random write, random read for 4g size and different block_size. Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- tests/iozone | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/iozone b/tests/iozone index 3095ce80..f0c2878c 100755 --- a/tests/iozone +++ b/tests/iozone @@ -7,7 +7,7 @@ for mnt in $mount_points do - cmd="iozone -a -f $mnt/iozonetest" + cmd="iozone -r $block_size -s 4g -i 0 -i 1 -i 2 -f $mnt/iozonetest" log_echo "$cmd &" $cmd > $TMP/iozone-$i & i=$((i+1)) -- 2.23.0

2 1

[PATCH lkp-tests 1/2] jobs/iozone.yaml: add specified block_size parameter
by Lu Kaiyi 22 Oct '20

22 Oct '20

[why] need to test different block_size value Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- jobs/iozone.yaml | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/jobs/iozone.yaml b/jobs/iozone.yaml index f281186b..d17137e8 100644 --- a/jobs/iozone.yaml +++ b/jobs/iozone.yaml @@ -1,13 +1,17 @@ suite: iozone testcase: iozone category: benchmark +block_size: +- 64k +- 256k +- 1m +- 4m +- 16m -disk: 2HDD +disk: 1HDD fs: -- xfs - ext4 -- btrfs iosched: - kyber -- 2.23.0

2 1

[PATCH lab-z9] hosts: add common configs
by Zhang Yu 22 Oct '20

22 Oct '20

Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- hosts/taishan200-2280-2s48p-512g | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 hosts/taishan200-2280-2s48p-512g diff --git a/hosts/taishan200-2280-2s48p-512g b/hosts/taishan200-2280-2s48p-512g new file mode 100644 index 0000000..5a4126a --- /dev/null +++ b/hosts/taishan200-2280-2s48p-512g @@ -0,0 +1,4 @@ +nr_cpu: 96 +nr_node: 4 +memory: 512G +arch: aarch64 -- 2.23.0

1 0

[PATCH compass-ci] git_mirror: put fork_stat to es
by Li Yuanchao 22 Oct '20

22 Oct '20

[why] We want to know whether git-mirror works well, so save last 10 times fetch_time and new_refs_time, and put them to es Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 37 +++++++++++++++++++++++++++++++------ 1 file changed, 31 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 593d1eb..2b78523 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -98,8 +98,10 @@ class MirrorMain @fork_stat[stat_key] = { queued: false, priority: 0, - fetch_time: nil, - new_refs_time: nil + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 } end @@ -152,10 +154,9 @@ class MirrorMain feedback_info = @feedback_queue.pop(true) git_repo = feedback_info[:git_repo] - @fork_stat[git_repo][:queued] = false + update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - @fork_stat[git_repo][:priority] += 1 return reload_fork_info if git_repo == 'upstream-repos/upstream-repos' new_refs = check_new_refs(git_repo) @@ -196,8 +197,8 @@ class MirrorMain @git_info[git_repo] = YAML.safe_load(File.open(repodir)) @git_info[git_repo]['git_repo'] = git_repo @git_info[git_repo].merge!(@defaults[project]) if @defaults[project] - es_repo_update(git_repo, @git_info[git_repo]) fork_stat_init(git_repo) + es_repo_update(git_repo) @priority_queue.push git_repo, @priority @priority += 1 end @@ -255,11 +256,35 @@ class MirrorMain end end - def es_repo_update(git_repo, repo_info) + def es_repo_update(git_repo) + repo_info = @git_info[git_repo].merge(@fork_stat[git_repo]) + repo_info.delete(:cur_refs) if repo_info.key?(:cur_refs) body = { "doc": repo_info, "doc_as_upsert": true } @es_client.update(index: 'repo', type: '_doc', id: git_repo, body: body) end + + def update_stat_fetch(git_repo) + @fork_stat[git_repo][:queued] = false + offset_fetch = @fork_stat[git_repo][:offset_fetch] + offset_fetch = 0 if offset_fetch >= 10 + @fork_stat[git_repo][:fetch_time][offset_fetch] = Time.now.to_s + @fork_stat[git_repo][:offset_fetch] = offset_fetch + 1 + end + + def update_stat_new_refs(git_repo) + @fork_stat[git_repo][:priority] += 1 + offset_new_refs = @fork_stat[git_repo][:offset_new_refs] + offset_new_refs = 0 if offset_new_refs >= 10 + @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s + @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 + end + + def update_fork_stat(git_repo, possible_new_refs) + update_stat_fetch(git_repo) + update_stat_new_refs(git_repo) if possible_new_refs + es_repo_update(git_repo) + end end -- 2.23.0

1 0

[PATCH v2 compass-ci] /src/lib/web_backend.rb: support multi-fields search in "/get_jobs"
by Zhang Yuhang 22 Oct '20

22 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 46 ++++++++++++++++++++++++------------------ 1 file changed, 26 insertions(+), 20 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..c3f6bef 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -248,23 +248,32 @@ def es_search(must, size, from) return es_query(query)['hits']['hits'], total end -def get_job(result) - job = {} - ALL_FIELDS.each do |f| - job[f] = result[f] +def get_jobs_result(result) + jobs = [] + result.each do |r| + job = {} + ALL_FIELDS.each do |f| + job[f] = r['_source'][f] + end + jobs << job end - job + jobs end -def search_job(git_repo, page_size, page_num) +def search_job(condition_fields, page_size, page_num) must = [] - must << { regexp: { upstream_repo: ".*#{git_repo}.*" } } if git_repo - jobs = [] - result, total = es_search(must, page_size, page_num * page_size) - result.each do |r| - jobs << get_job(r['_source']) + FIELDS.each do |field| + value = space_to_nil(condition_fields[field]) + next unless value + + must << if field.to_s == 'upstream_repo' + { regexp: { field => ".*#{value}.*" } } + else + { term: { field => value } } + end end - return jobs, total + result, total = es_search(must, page_size, page_num * page_size) + return get_jobs_result(result), total end def get_banner(git_repo, branches) @@ -291,17 +300,14 @@ def get_optimize_jobs_braches(jobs) end def get_jobs_body(params) - git_repo = space_to_nil(params[:upstream_repo]) - page_size = get_positive_number(params[:page_size], 20) - page_num = get_positive_number(params[:page_num], 1) - 1 - jobs, total = search_job(git_repo, page_size, page_num) + page_size = get_positive_number(params.delete(:page_size), 20) + page_num = get_positive_number(params.delete(:page_num), 1) - 1 + jobs, total = search_job(params, page_size, page_num) jobs, branches = get_optimize_jobs_braches(jobs) { total: total, - filter: { - upstream_repo: git_repo - }, - banner: get_banner(git_repo, branches), + filter: params, + banner: get_banner(params[:upstream_repo], branches), jobs: jobs, fields: FIELDS }.to_json -- 2.23.0

1 0

[PATCH v4 lkp-tests] sshd: send email to users for login services
by Zhang Yale 22 Oct '20

22 Oct '20

[why] Provide a login port for users by email, determine whether to send the ssh_port message based on the "pub_key". Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 99 +++++++++++++++++++++++++++++++++++++++-------------- 1 file changed, 74 insertions(+), 25 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..82db61a6 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key -# email +# my_email +# sshr_port +# sshr_port_len +# sshr_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +[ -n "$sshr_port" ] || sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=50000 run_ssh() { @@ -11,40 +21,79 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { - deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh - -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: - - Login: - ssh root@$PUB_IP - Due time: - $deadline - - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch + data="To: $my_email +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch Regards -compass-ci +Compass-Ci " } +data_failure() +{ + data="To: $my_email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $my_username: + Sorry to inform you that your application failed. + You may need to wait a moment, or check whether your pub_key exists. + +Regards +Compass-Ci +" +} + +compose_email() +{ + deadline=$(date -d "$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +set_port() +{ + [ -n "$pub_key" ] || return + + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] && return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$my_email" ] || return 0 + my_username=$(echo "$my_email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +set_port +report_ssh_port $port run_email -- 2.23.0

1 0

[PATCH v3 lkp-tests] sshd: send emails to login services for users
by Zhang Yale 22 Oct '20

22 Oct '20

[why] Provide a login port for users by email, determine whether to send the ssh_port message based on the "pub_key". Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- daemon/sshd | 98 ++++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 74 insertions(+), 24 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 98f31d77..1ebc9d00 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -1,6 +1,16 @@ #!/bin/sh # pub_key -# email +# my_email +# sshr_port +# sshr_port_len +# sshr_port_base + +. $LKP_SRC/lib/http.sh + +sshr_ip=$LKP_SERVER +[ -n "$sshr_port" ] || sshr_port=5050 +[ -n "$sshr_port_len" ] || sshr_port_len=2000 +[ -n "$sshr_port_base" ] || sshr_port_base=50000 run_ssh() { @@ -11,40 +21,80 @@ run_ssh() systemctl start sshd } -compose_email() +data_success() { deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") - data="To: $email -Subject: [NOTIFY compass-ci] $testbox ready to ssh - -Dear $email: - Thanks for your participation in Kunpeng and software ecosystem! - According to your application, $testbox has been provisioned. - The datails are as follows: - - Login: - ssh root@$PUB_IP - Due time: - $deadline - - HW: - nr_cpu: $nr_cpu - memory: $memory - testbox: $testbox - OS: - $os $os_version $os_arch + data="To: $my_email +Subject: [NOTIFY Compass-ci] $testbox ready to use + +Dear $my_username: + Thanks for your participation in software ecosystem! + According to your application, $testbox has been provisioned. + The datails are as follows: + + Login: + ssh root(a)api.compass-ci.openeuler.org -p $port + Due time: + $deadline + HW: + nr_cpu: $nr_cpu + memory: $memory + testbox: $testbox + OS: + $os $os_version $os_arch + +Regards +Compass-Ci +" +} + +data_failure() +{ + data="To: $my_email +Subject: [NOTIFY Compass-ci] Applying $testbox failed + +Dear $my_username: + Sorry to inform you that your application failed. + You may need to wait a moment, or check whether your pub_key exists. Regards -compass-ci +Compass-Ci " } +compose_email() +{ + deadline=$(date -d "+$runtime seconds" +"%Y-%m-%d %H:%M:%S") + if [ -n "$port" ]; then + data_success + else + data_failure + fi +} + +set_port() +{ + [ -n "$pub_key" ] || return + + for i in $(seq 1 10) + do + port=$(($(date +%s%N)%"$sshr_port_len"+"$sshr_port_base")) + ssh -o StrictHostKeyChecking=no -o ExitOnForwardFailure=yes -o TCPKeepAlive=yes \ + -Nf -R $port:localhost:22 sshr@"$sshr_ip" -p "$sshr_port" &>/dev/null + [ $? -eq 0 ] && return + done + port="" +} + run_email() { - [ -n "$email" ] || return 0 + [ -n "$my_email" ] || return 0 + my_username=$(echo "$my_email" | awk -F '@' '{print $1}') compose_email - curl -XPOST "$MAIL_HOST:$MAIL_PORT/send_mail_text" -d "$data" + curl -XPOST "${MAIL_HOST:-$LKP_SERVER}:${MAIL_PORT:-11312}/send_mail_text" -d "$data" } run_ssh +set_port +report_ssh_port $port run_email -- 2.23.0

2 2

[PATCH compass-ci] lib/sched.cr: add result_root output logs
by Wu Zhende 22 Oct '20

22 Oct '20

[Why] The result_root directory of the job stores the running results. It plays an important role in locating problems. So puts this info to docker logs, submit -m can show result_root. [Output Example] {"job_id"=>"z9.1", "message"=>"", "job_state"=>"submit", "result_root"=>"/srv/result/iperf/vm-2p8g/2020-10-22/z9.1"} {"job_id"=>"z9.1", "result_root"=>"/srv/result/iperf/vm-2p8g/2020-10-22/z9.1", "job_state"=>"set result root"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 7215df5..070a16b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -227,6 +227,7 @@ class Sched "job_id" => job_id, "message" => message.to_s, "job_state" => "submit", + "result_root" => "/srv#{job.result_root}" } return job_messages if response["error"]? end @@ -278,6 +279,7 @@ class Sched "job_id" => job_id, "message" => message.to_s, "job_state" => "submit", + "result_root" => "/srv#{job.result_root}" }] end @@ -441,6 +443,7 @@ class Sched if job job.update({"testbox" => testbox}) job.set_result_root + puts %({"job_id": "#{job_id}", "result_root": "/srv#{job.result_root}", "job_state": "set result root"}) @redis.set_job(job) end return job -- 2.23.0

1 0