- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] fix: sched: set 2 fields to job: direct_macs, direct_ips
by Ren Wen 21 Oct '20

21 Oct '20

[Why] nodes need communicate with each other when running cluster jobs. after adding 2 fileds, 'direct_ips' and 'direct_macs', can bind ips to macs. 'macs' get from cluster file ('$LKP_SRC/cluster/*'), 'ips' set by scheduler default: 192.168.222.<1..250> Now use 192.168.222.<1..250> to test, In future, use DHCP sever interface to get direct_ips. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/sched.cr | 30 +++++++++++++++++++++++++++++- 1 file changed, 29 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 7215df5..d669c50 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -147,6 +147,25 @@ class Sched return YAML.parse(File.read(cluster_file_path)).as_h end + # a private-ip pools, each return 10 ips + private def get_private_ips + # ip = net_id + host_id + # host_id : 1..250 + # will divide into 10 groups used by maximum 10 cluster jobs + host_ids = [] of Int32 + 25.times do |i| + host_ids << i * 10 + 1 + end # [1, 11, ..., 241] + net_id = "192.168.222" + index = @redis.get_seqno % 25 + host_id = host_ids[index] + ips = [] of String + 10.times do |i| + ips << "#{net_id}.#{host_id + i}" + end + ips + end + def get_commit_date(job) if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", @@ -198,6 +217,9 @@ class Sched # collect all job ids job_ids = [] of String + # an array consists of 10 ips + ips = get_private_ips + # steps for each host cluster_config.each do |host, config| tbox_group = host.to_s @@ -219,7 +241,13 @@ class Sched job["testbox"] = tbox_group job.update_tbox_group(tbox_group) job["node_roles"] = config["roles"].as_a.join(" ") - job["node_macs"] = config["macs"].as_a.join(" ") + direct_macs = config["macs"].as_a + direct_ips = [] of String + direct_macs.size.times do + direct_ips << ips.pop {raise "more than 10 macs"} + end + job["direct_macs"] = direct_macs.join(" ") + job["direct_ips"] = direct_ips.join(" ") response = add_job(job, job_id) message = (response["error"]? ? response["error"]["root_cause"] : "") -- 2.23.0

2 2

[PATCH compass-ci 2/2] feat: redis: add a function to get a seq number
by Ren Wen 21 Oct '20

21 Oct '20

[Why] Need to allocate ips to nodes when running a cluster job. A seq number 'n' used to divide ip regions, to avoid ip conflict when more than 1 cluster jobs running at same time. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/redis_client.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/redis_client.cr b/src/scheduler/redis_client.cr index b816d67..7675056 100644 --- a/src/scheduler/redis_client.cr +++ b/src/scheduler/redis_client.cr @@ -28,6 +28,10 @@ class Redis::Client @client.hdel(key, field.to_s) end + def get_seqno + @client.incr("sched/seqno") + end + def get_job(job_id : String) job_hash = @client.hget("sched/id2job", job_id) if !job_hash -- 2.23.0

1 1

[PATCH compass-ci] feat: taskqueue: support query keys
by Tong Qunfeng 21 Oct '20

21 Oct '20

[why] support multi-queue-name for [tbox_group, host, ...] scheduler need know the sub-queue for it. like: "sched/$tbox_group-1" "sched/$tbox_group/1024" "sched/$tbox_group.taishan200-2280..." [how] - taskqueue support "/keys" API. - scheduler use @task_queue.query_keys("sched/$tbox_group*") to get the keys list. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/taskqueue_api.cr | 8 ++++++++ src/taskqueue/queue.cr | 12 +++++++++++- src/taskqueue/redis_client.cr | 22 ++++++++++++++++++++++ src/taskqueue/taskqueue.cr | 15 +++++++++++++++ 4 files changed, 56 insertions(+), 1 deletion(-) diff --git a/src/lib/taskqueue_api.cr b/src/lib/taskqueue_api.cr index 99c3933..5a83a15 100644 --- a/src/lib/taskqueue_api.cr +++ b/src/lib/taskqueue_api.cr @@ -18,6 +18,14 @@ class TaskQueueAPI arrange_response(response) end + def query_keys(service_key_with_wild_char : String) + params = HTTP::Params.encode({"queue" => service_key_with_wild_char}) + client = HTTP::Client.new(@host, port: @port) + response = client.get("/keys?" + params) + client.close + arrange_response(response) + end + def consume_task(service_queue_path : String) params = HTTP::Params.encode({"queue" => service_queue_path}) response_put_api("consume", params) diff --git a/src/taskqueue/queue.cr b/src/taskqueue/queue.cr index 674606e..f98ee06 100644 --- a/src/taskqueue/queue.cr +++ b/src/taskqueue/queue.cr @@ -146,6 +146,16 @@ class TaskQueue end end + def queue_respond_keys(env) + queue_name_pattern, ext_set = queue_check_params(env, ["queue"]) + return ext_set if ext_set + + response = get_keys(queue_name_pattern[0]) + env.response.status_code = 201 if response.empty? + + return response + end + # loop try: when there has no task, return until get one or timeout # default timeout is 0.015ms, we delay for 10ms at each time private def operate_with_timeout(timeout) @@ -195,7 +205,7 @@ class TaskQueue # prefix is start with "queues", need delete it s_name = prefix.sub("#{QUEUE_NAME_BASE}/", "") move_first_task_in_redis_with_score("#{s_name}/#{uuid}", - "#{s_name}/ready") + "#{s_name}/ready") end end end diff --git a/src/taskqueue/redis_client.cr b/src/taskqueue/redis_client.cr index 27d46cf..7f8ea59 100644 --- a/src/taskqueue/redis_client.cr +++ b/src/taskqueue/redis_client.cr @@ -261,6 +261,28 @@ class TaskQueue return keys end + private def get_keys(queue_name) + query_prefix = QUEUE_NAME_BASE + "/" + query_prefix_len = query_prefix.size + search = query_prefix + queue_name + response = [] of String + + cursor = "0" + loop do + cursor, keys = @redis.scan(cursor, search, 256) + case keys + when Array(Redis::RedisValue) + keys.each do |key| + response << "#{key}"[query_prefix_len..-1] + end + end + + break if cursor == "0" + end + + return response + end + private def move_first_task_in_redis_with_score(from : String, to : String) # result was ["crystal.87230", "1600782938.9017849"] result = @redis.zrange("#{QUEUE_NAME_BASE}/#{from}", 0, 0, with_scores: true) diff --git a/src/taskqueue/taskqueue.cr b/src/taskqueue/taskqueue.cr index 98c5623..13f48db 100644 --- a/src/taskqueue/taskqueue.cr +++ b/src/taskqueue/taskqueue.cr @@ -104,6 +104,21 @@ class TaskQueue nil end + # ------------------- + # request: curl http://localhost:3060/keys? + # queue=sched* + # wild match: *, ?, [-] + # + # response: 200 ["scheda", "schedb", ...] + # 201 ## when no find + # 400 "Missing parameter <queue>" + # 413 "Query results too large keys" + get "/keys" do |env| + response = queue_respond_keys(env) + # debug_message(env, response, Time.utc) # maybe too large + response.to_json unless env.response.status_code == 201 + end + @port = (ENV.has_key?("TASKQUEUE_PORT") ? ENV["TASKQUEUE_PORT"].to_i32 : TASKQUEUE_PORT) Kemal.run(@port) end -- 2.23.0

1 0

[PATCH lkp-tests] lib/job2sh.rb: fix monitors run twice
by Wei Jihui 21 Oct '20

21 Oct '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job2sh.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 75e28ef6..9c95ff4b 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files'].include?(key) + if ['pp', 'define_files', 'monitors'].include?(key) return false end tabs = indent(ancestors) -- 2.23.0

1 0

[PATCH compass-ci 3/3] user-client/maintain: mail rootfs test report
by Liu Yinsi 21 Oct '20

21 Oct '20

use send-mail service to mail test report, data line must be at the top, or mail failed. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- .../maintain/walk-os-test/walk-os-iperf-test | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/user-client/maintain/walk-os-test/walk-os-iperf-test b/user-client/maintain/walk-os-test/walk-os-iperf-test index ccbff68..91a3541 100755 --- a/user-client/maintain/walk-os-test/walk-os-iperf-test +++ b/user-client/maintain/walk-os-test/walk-os-iperf-test @@ -45,6 +45,20 @@ TESTBOX = %w[vm-2p8g vm-pxe-2p8g].freeze TEST_REPORT = 'walk-test.report' TEST_YAML = 'iperf-walk-os.yaml' +def mail_report(data) + %x(curl -X POST 'localhost:11312/send_mail_yaml' -d "#{data}") +end + +def get_mail_data + mail_body = File.read(TEST_REPORT).chomp + data = " +subject: os rootfs test report +to: user@emailaddress +body: '#{mail_body}'" + + mail_report(data) +end + def logging(*log_params) log_format = '%-18s %-37s %-23s %-23s %-12s %-13s %-50s' File.open(TEST_REPORT, 'a') do |f| @@ -114,3 +128,5 @@ end logging('Job_ID', 'OS', 'Begin', 'End', 'Cost/sec', 'Job_State', 'Result_Url') MOUNT_TYPE.each { |os_mount| get_testbox(os_mount) } + +get_mail_data -- 2.23.0

2 4

[PATCH v6 compass-ci] feat: build centos initramfs image with one command
by Wang Chenglong 21 Oct '20

21 Oct '20

Use the tool to create a new centos-$version initramfs image with one command. and we also can customize the pre-installed software by configuring ./bin/packages-to-install. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/centos/README.md | 15 ++++++++ container/osimage/centos/create-image | 37 ++++++++++++++++++++ container/osimage/centos/files-to-exclude | 11 ++++++ container/osimage/centos/lib | 37 ++++++++++++++++++++ container/osimage/centos/packages-to-install | 9 +++++ container/osimage/centos/run | 27 ++++++++++++++ 6 files changed, 136 insertions(+) create mode 100644 container/osimage/centos/README.md create mode 100755 container/osimage/centos/create-image create mode 100644 container/osimage/centos/files-to-exclude create mode 100755 container/osimage/centos/lib create mode 100644 container/osimage/centos/packages-to-install create mode 100755 container/osimage/centos/run diff --git a/container/osimage/centos/README.md b/container/osimage/centos/README.md new file mode 100644 index 0000000..88166c8 --- /dev/null +++ b/container/osimage/centos/README.md @@ -0,0 +1,15 @@ +# Use the tool to create a new centos-${os_version} initramfs image. + +Usage: + cd ${CCI_SRC}/rootfs/initramfs/centos/aarch64/${os_version}/ + ./build + +Some configuration items: +./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + +./files-to-exclude + If you want remove some unnecessary files, you can write the names in ./files-to-exclude + +$HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. diff --git a/container/osimage/centos/create-image b/container/osimage/centos/create-image new file mode 100755 index 0000000..a6a2766 --- /dev/null +++ b/container/osimage/centos/create-image @@ -0,0 +1,37 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +install_packages() +{ + yum repolist + yum -y install --skip-broken $(</tmp/packages-to-install) + yum clean all +} + +setup_rootfs() +{ + ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime +} + + +setup_login() { + [ -n "$ROOT_PASSWD" ] || return + echo "Changing root password" + passwd_md5=$(openssl passwd -1 "$ROOT_PASSWD") + sed -i -r "s/^root:[^:]*:(.*)/root:${passwd_md5//\//\\/}:\1/" "$ROOTFS_DIR/etc/shadow" + + sed -i 's/[# ]PermitRootLogin.*/PermitRootLogin yes/' "$ROOTFS_DIR/etc/ssh/sshd_config" +} + +pack_cgz() +{ + echo "Packing package. Please wait." + find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + chmod 644 /tmp/${IMAGE_NAME} +} + +install_packages +setup_rootfs +setup_login +pack_cgz diff --git a/container/osimage/centos/files-to-exclude b/container/osimage/centos/files-to-exclude new file mode 100644 index 0000000..3854947 --- /dev/null +++ b/container/osimage/centos/files-to-exclude @@ -0,0 +1,11 @@ +/.dockerenv +/lib/modules +/usr/lib/modules +/usr/share/doc +/usr/share/man +/usr/share/info +/usr/share/i18n +/usr/share/locale +/usr/share/terminfo +/tmp/files-to-exclude +/tmp/packages-to-install diff --git a/container/osimage/centos/lib b/container/osimage/centos/lib new file mode 100755 index 0000000..5bf1463 --- /dev/null +++ b/container/osimage/centos/lib @@ -0,0 +1,37 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +DIR="$(pwd)" +os_name="$(echo $DIR |awk -F "/" '{print $(NF - 2)}')" +os_version="$(echo $DIR |awk -F "/" '{print $NF}')" +image="${os_name}:${os_version}" +IMAGE_NAME="${os_name}-${os_version}-$(date +"%Y%m%d").cgz" + +export_root_passwd() { + local root_pwd_file="$HOME/.config/compass-ci/rootfs-passwd" + [ -f "$root_pwd_file" ] || { + echo "[INFO] Please set the password file." + echo "$HOME/.config/compass-ci/rootfs-passwd" + exit 1 + } + + export ROOT_PASSWD=$(cat "$root_pwd_file") +} + +pull_docker_image() +{ + docker pull $image + if [ $? = 0 ]; then + echo "finish downloading image" + else + echo "[ERROR] Pull docker image failed." + exit 1 + fi +} + +docker_cp_image() +{ + docker cp -a init_docker:/tmp/${IMAGE_NAME} $HOME/ + echo "result: $(ls $HOME/${IMAGE_NAME})" +} diff --git a/container/osimage/centos/packages-to-install b/container/osimage/centos/packages-to-install new file mode 100644 index 0000000..a09bc28 --- /dev/null +++ b/container/osimage/centos/packages-to-install @@ -0,0 +1,9 @@ +openssh-server +openssl +cifs-utils +hostname +curl +iproute +iputils +which +time diff --git a/container/osimage/centos/run b/container/osimage/centos/run new file mode 100755 index 0000000..51220c6 --- /dev/null +++ b/container/osimage/centos/run @@ -0,0 +1,27 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. ${CCI_SRC}/container/osimage/centos/lib +. ${CCI_SRC}/container/defconfig.sh + +export_root_passwd + +DIR=$(dirname $(realpath $0)) +cmd=( + docker run + --name init_docker + -v $DIR/create-image:/root/bin/create-image + -v $DIR/packages-to-install:/tmp/packages-to-install + -v $DIR/files-to-exclude:/tmp/files-to-exclude + -e ROOT_PASSWD=$ROOT_PASSWD + -e IMAGE_NAME=$IMAGE_NAME + $image + /root/bin/create-image +) + +pull_docker_image +${cmd[@]} +docker_cp_image +docker_rm init_docker &> /dev/null +echo "build finished" -- 2.23.0

2 1

[PATCH lkp-tests] lib: add commment in add_pp and add_monitors
by Hu Xuejiao 21 Oct '20

21 Oct '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- lib/job.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index e2416eb8..f24435a0 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -565,6 +565,7 @@ class Job end def add_monitors + @job[comment_to_symbol('auto generated by add_monitors()')] = nil @job['monitors'] = {} for_each_in(@job, monitor_params) do |_pk, _h, m_n, _m_args| @job['monitors'][m_n] = nil @@ -572,6 +573,7 @@ class Job end def add_pp() + @job[comment_to_symbol('auto generated by add_pp()')] = nil @job["pp"] = Hash.new() init_program_options() for_each_in(@job, @referenced_programs.keys) do |_pk, h, p_n, p_args| -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/deploy-cci: determine whether GIT_SERVER exists
by Cui Yili 21 Oct '20

21 Oct '20

Determine whether GIT_SERVER exists, if not, there is no need to convert the git address. Signed-off-by: cuiyili <2268260388(a)qq.com> --- tests/deploy-cci | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index edfc5540..e296568f 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -3,10 +3,12 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # redirect remote-repository address to "/srv/git/gitee.com" -cat >> /etc/gitconfig <<EOF -[url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com -EOF +if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF +fi export initrd_http_host=$INITRD_HTTP_HOST export initrd_http_port=$INITRD_HTTP_PORT -- 2.23.0

1 0

[PATCH lkp-tests] tests/deploy-cci: determine whether GIT_SERVER exists
by Cui Yili 21 Oct '20

21 Oct '20

Determine whether GIT_SERVER exists, if not, there is no need to convert the git address. Signed-off-by: cuiyili <2268260388(a)qq.com> --- tests/deploy-cci | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index edfc5540..a7bce678 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -3,10 +3,12 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # redirect remote-repository address to "/srv/git/gitee.com" -cat >> /etc/gitconfig <<EOF -[url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com -EOF +if [ -n "$GIT_SERVER" ];then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF +fi export initrd_http_host=$INITRD_HTTP_HOST export initrd_http_port=$INITRD_HTTP_PORT -- 2.23.0

2 1

[PATCH v3 compass-ci] /src/lib/web_backend.rb: fix "/compare_candidates" tbox_group regex bug
by Zhang Yuhang 21 Oct '20

21 Oct '20

Info: "/compare_canditates" error, due to fault regular expression Detail: Miss a couple brackets, so will match /\d+$/. Error Flow: 1. like "xxx123" will be matched 2. index = "xxx123".index('--') || "xxx123".rindex('-') 3. index equal "nil" 4. r = r[0, nil] will raise an error "TypeError (no implicit conversion from nil to integer)" Solution: Fill a couple brackets, so will match "vm-abc-0abc-abc" or "vm-abc-0abc-012". Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- src/lib/web_backend.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 48cfadb..fda01ca 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -47,7 +47,7 @@ end def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-([a-zA-Z]+)|(\d+)$)/ + if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ index = r.index('--') || r.rindex('-') r = r[0, index] end -- 2.23.0

3 5

2024

2023

2022

2021

2020

Compass-ci