March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] lib/lifecycle.cr: don't update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

Change the logic [Orginal] Update the testbox time in the ES database when reciving the job events [Now] Don't update in the ES. Only update the cache. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 25 ++++++++++--------------- 1 file changed, 10 insertions(+), 15 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index 3a72906..16830d0 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -144,21 +144,17 @@ class Lifecycle def update_cached_machine(testbox, event) machine = @machines[testbox]? - return if machine && !event["time"].to_s.bigger_than?(machine["time"]?) + if machine + return unless event["time"].to_s.bigger_than?(machine["time"]?) - update_es_machine_time(testbox, event) - end - - def update_es_machine_time(testbox, event) - machine = @es.get_tbox(testbox) - return unless machine - return unless event["time"].to_s.bigger_than?(machine["time"]?) + machine.as_h["time"] = event["time"] + else + machine = @es.get_tbox(testbox) + return unless machine - machine.as_h.delete("history") - machine.as_h["time"] = event["time"] - machine.as_h["state"] = JSON::Any.new("booting") - @machines[testbox] = machine - @es.update_tbox(testbox, machine.as_h) + machine.as_h.delete("history") + @machines[testbox] = machine + end end def update_cached_job(job_id, event) @@ -187,8 +183,7 @@ class Lifecycle def on_job_close(event) event_job_id = event["job_id"].to_s - job = @jobs[event_job_id] - + job = @jobs[event_job_id]? return unless job @jobs.delete(event_job_id) -- 2.23.0

1 0

[PATCH compass-ci 1/2] service/scheduler: update testbox's time in es
by Wu Zhende 08 Mar '21

08 Mar '21

When the machine accesses the API of the scheduler to update the job status, the machine's active time is also updated. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/sched.cr | 7 +++++++ src/scheduler/update_job_parameter.cr | 4 +++- 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 51f493e..f5aba03 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -163,6 +163,13 @@ class Sched @es.update_tbox(testbox.to_s, hash) end + def update_testbox_time(job_id) + job = @redis.get_job(job_id) + testbox = job["testbox"] + hash = {"time" => @env.get?("time").to_s} + @es.update_tbox(testbox, hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/update_job_parameter.cr b/src/scheduler/update_job_parameter.cr index cc4cf87..b43aa06 100644 --- a/src/scheduler/update_job_parameter.cr +++ b/src/scheduler/update_job_parameter.cr @@ -9,6 +9,7 @@ class Sched end @env.set "job_id", job_id + @env.set "time", get_time # try to get report value and then update it job_content = {} of String => String @@ -34,13 +35,14 @@ class Sched @log.info(log.to_json) @env.set "job_state", job_content["job_state"]? + update_testbox_time(job_id) rescue e @log.warn(e) ensure mq_msg = { "job_id" => @env.get?("job_id").to_s, "job_state" => (@env.get?("job_state") || "update").to_s, - "time" => get_time + "time" => @env.get?("time").to_s } @mq.pushlish_confirm(JOB_MQ, mq_msg.to_json) end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sparrow/0-package/read-config: export config yaml info
by Liu Yinsi 08 Mar '21

08 Mar '21

export my_email, my_name, server_ip to deploy compass-ci. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/read-config | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100755 sparrow/0-package/read-config diff --git a/sparrow/0-package/read-config b/sparrow/0-package/read-config new file mode 100755 index 0000000..c664e30 --- /dev/null +++ b/sparrow/0-package/read-config @@ -0,0 +1,14 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# export config info setup.yaml server_ip, my_email, my_name. + +mkdir -p /etc/compass-ci/ +cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml + +options=( server_ip my_name my_email ) + +for option in ${options[@]} +do + export $option=$(awk -F ": " '/^'$option'/ {print $2}' /etc/compass-ci/setup.yaml) +done -- 2.23.0

1 1

[PATCH v2 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script
by Liu Yinsi 08 Mar '21

08 Mar '21

deploy compass-ci client, provides function: submit job to compass-ci server and boot testbox. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-client | 71 ++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 71 insertions(+) create mode 100755 sparrow/install-client diff --git a/sparrow/install-client b/sparrow/install-client new file mode 100755 index 0000000..b077a71 --- /dev/null +++ b/sparrow/install-client @@ -0,0 +1,71 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +[[ $CCI_SRC ]] || export CCI_SRC=$(git rev-parse --show-toplevel) + +. $CCI_SRC/lib/log.sh + +check_server_ip() +{ + [ -z "$server_ip" ] && . 0-package/read-config + [ -z "$server_ip" ] && { + log_error "ENV \$server_ip doesn't exist, please config $CCI_SRC/sparrow/setup.yaml" + exit 1 + } +} + +install_env() +{ + cd /c/compass-ci/sparrow || return + check_server_ip + 0-package/install + 1-storage/tiny + 5-build/ipxe & + 1-storage/permission + 2-network/br0 + 2-network/iptables + 3-code/git + 3-code/dev-env + . /etc/profile.d/compass.sh + 5-build/register-account +} + +boot_ipxe() +{ + SCHED_HOST=$server_ip + sed -i "s%172.17.0.1%${SCHED_HOST}%g" /tftpboot/boot.ipxe + sed -i "s%3000%${SCHED_PORT:-3000}%g" /tftpboot/boot.ipxe +} + +run_service() +{ + ( + cd $CCI_SRC/container/dnsmasq || return + ./build + ./start + boot_ipxe + )& + ( + cd $CCI_SRC/container/qemu-efi || return + ./build + ./install + )& + ( + cd $CCI_SRC/container/fluentd-base || return + ./build + + cd $CCI_SRC/container/sub-fluentd || return + ./build + ./start + )& +} + +main() +{ + install_env + run_service +} + +main +wait -- 2.23.0

1 1

[PATCH compass-ci 1/5] container/dracut-initrd: add lkp module into conf file
by Xu Xijian 08 Mar '21

08 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/conf/add-modules.conf | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/conf/add-modules.conf b/container/dracut-initrd/conf/add-modules.conf index b8048a1..6601277 100644 --- a/container/dracut-initrd/conf/add-modules.conf +++ b/container/dracut-initrd/conf/add-modules.conf @@ -1 +1 @@ -add_dracutmodules+=" cifs " +add_dracutmodules+=" cifs lkp " -- 2.23.0

1 0

[PATCH v2 lkp-tests] setup/simplify-ci: call $CCI_SRC/sparrow/install-client
by Liu Yinsi 08 Mar '21

08 Mar '21

move simplify-ci main function to $CCI_SRC/sparrow/install-client script, make code more reusablely. for simplicy-ci job test, if there is $SCHED_HOST, and export server_ip=$SCHED_HOST, when call $CCI_SRC/sparrow/install-client, no need to config sparrow/setup.yaml, and also no need to execute sparrow/0-package/read-config(export server_ip). for locally deploy compass-ci client, must config sparrow/setup.yaml, and also need to execute sparrow/0-package/read-config(export server_ip), if server_ip not exists, will call log_error() and exit. refer patch: Subject: [PATCH v2 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 61 ++++------------------------------------------- 1 file changed, 5 insertions(+), 56 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 5aede9cdb..107cef2e8 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +export server_ip=$SCHED_HOST git_ci() { @@ -17,61 +16,11 @@ git_ci() git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } -dev_env() +deploy() { - export sched_host=$SCHED_HOST - export sched_port=$SCHED_PORT - 3-code/dev-env + cd /c/compass-ci/sparrow && ./install-client } -install_env() -{ - cd /c/compass-ci/sparrow || return - 0-package/install - 1-storage/tiny - 5-build/ipxe & - 1-storage/permission - 2-network/br0 - 2-network/iptables - 3-code/git - dev_env - . /etc/profile.d/compass.sh -} - -boot_ipxe() -{ - sed -i "s%172.17.0.1%$SCHED_HOST%g" /tftpboot/boot.ipxe - sed -i "s%3000%$SCHED_PORT%g" /tftpboot/boot.ipxe -} - -run_service() -{ - ( - cd $CCI_SRC/container/dnsmasq || return - ./build - ./start - boot_ipxe - )& - ( - cd $CCI_SRC/container/qemu-efi || return - ./build - ./install - )& - ( - cd $CCI_SRC/container/fluentd-base || return - ./build - cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start - )& -} - -main() -{ - git_ci || return 1 - install_env - run_service -} +git_ci || return 1 +deploy -main -wait -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: optimize the get_pkg_common_dir function
by Li Ping 08 Mar '21

08 Mar '21

[why] 1. rpm has no much to do with os_mount. add a judgment 2. optimize the code style Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..5365bb2 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -258,23 +258,28 @@ class Job end def get_pkg_common_dir - pkg_style = nil + pkg_style = Hash(String, JSON::Any).new {|h, k| h[k] = JSON::Any.new(nil)} ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| - pkg_style = @hash[item]? - break if pkg_style - end - return nil unless pkg_style + next unless @hash[item]? - pkg_style = JSON.parse("{}") if pkg_style == nil + pkg_style.merge!((a)hash[item].as_h) + break unless pkg_style.empty? + end + return nil if pkg_style.empty? - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + tmp_os = pkg_style["os"].as_s? || "#{os}" + tmp_os_arch = pkg_style["os_arch"].as_s? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"].as_s? || "#{os_version}" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + mount_type = pkg_style["os_mount"].as_s? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

1 0

[PATCH lab-z9] jobs: add sysbench-mysql.yaml
by Wei Jihui 08 Mar '21

08 Mar '21

sysbench-mysql.yaml use lab-z9' cluster:cs-s1-a102-c1 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/sysbench-mysql.yaml | 54 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 54 insertions(+) create mode 100644 jobs/sysbench-mysql.yaml diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml new file mode 100644 index 0000000..792645c --- /dev/null +++ b/jobs/sysbench-mysql.yaml @@ -0,0 +1,54 @@ +suite: sysbench-mysql +category: benchmark + +cluster: cs-s1-a102-c1 +kernel_version: 4.19.90-mysql +kernel_custom_params: sched_steal_node_limit=8 + +mysql_user: +mysql_host: +mysql_port: +mysql_db: +mysql_password: + +if role server: + sysctl: + sc_numa_balancing: 0 + sc_sched_autogroup_enabled: 0 + sc_sched_min_granularity_ns: 10000000 + sc_sched_wakeup_granularity_ns: 15000000 + sc_threads-max: 52130 + sc_tcp_max_syn_backlog: 360000 + sc_dirty_ratio: 5 + sc_somaxconn: 1024 + sc_rmem_max: 16777216 + sc_wmem_max: 16777216 + sc_swappiness: 1 + sc_ipv4_tcp_rmem: 4096 87380 16777216 + sc_ipv4_tcp_wmem: 4096 87380 16777216 + sc_sched_features: NO_WAKEUP_PREEMPTION + iosched: deadline + iosched_dev: nvme0n1 + block_queue: + nr_requests: 2048 + block_queue_dev: nvme0n1 + nic_irp: + nics: enp189s0f1 + core_ranges: "55-79,93-95" + mysql: + data_dev: nvme0n1 + +if role client: + sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 600 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: false + report_interval: 1 + events: 2000000000 -- 2.23.0

1 0

[PATCH lkp-tests] tests/sysbench-mysql: accelerate data preparation
by Wei Jihui 08 Mar '21

08 Mar '21

1. accelerate data preparation from 20+min to 5+min 2. disable useless output Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- tests/sysbench-mysql | 25 +++++++++---------------- 1 file changed, 9 insertions(+), 16 deletions(-) diff --git a/tests/sysbench-mysql b/tests/sysbench-mysql index 615bc6e7..0ade2dab 100755 --- a/tests/sysbench-mysql +++ b/tests/sysbench-mysql @@ -1,9 +1,6 @@ #!/bin/sh -# - oltp_test_mode # - oltp_tables_count # - oltp_table_size -# - max_requests -# - mysql_table_engine # - nr_threads # - rand_type # - rand_spec_pct @@ -21,14 +18,11 @@ : "${mysql_user:=root}" : "${mysql_host:=$server}" : "${mysql_port:=3306}" -: "${mysql_db:=sysbench_1}" +: "${mysql_db:=sysbench_test}" : "${mysql_password:=mysql_password}" : "${db_driver:=mysql}" -: "${oltp_test_mode:=complex}" : "${oltp_tables_count:=1000}" : "${oltp_table_size:=100000}" -: "${max_requests:=0}" -: "${mysql_table_engine:=innodb}" : "${rand_type:=special}" : "${rand_spec_pct:=100}" : "${nr_threads:=256}" @@ -42,13 +36,10 @@ args1=( --mysql-host=$mysql_host --mysql-port=$mysql_port --mysql-db=$mysql_db - --oltp-test-mode=$oltp_test_mode --db-driver=$db_driver --mysql-password=$mysql_password - --max-requests=$max_requests - --mysql-table-engine=$mysql_table_engine - --oltp-table-size=$oltp_table_size - --oltp-tables-count=$oltp_tables_count + --table-size=$oltp_table_size + --tables=$oltp_tables_count --rand-type=$rand_type --rand-spec-pct=$rand_spec_pct --threads=$nr_threads @@ -72,8 +63,8 @@ args2=( stop_firewalld() { - systemctl stop firewalld - iptables -F + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 } run_sysbench_step() @@ -85,9 +76,11 @@ run_sysbench_step() run_sysbench_mysql() { - run_sysbench_step parallel_prepare.lua ${args1[@]} prepare + sysbench ${args1[@]} oltp_read_write prepare > /dev/null + run_sysbench_step oltp.lua ${args2[@]} run - run_sysbench_step oltp.lua ${args2[@]} cleanup + + run_sysbench_step oltp.lua ${args2[@]} cleanup > /dev/null } stop_firewalld -- 2.23.0

1 0

[PATCH v3 lkp-tests] /stats add proc-vmstat.rb
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] convert stats/proc-vmstat to proc_vmstat() of proc-vmstat.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/proc-vmstat.rb | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 stats/proc-vmstat.rb diff --git a/stats/proc-vmstat.rb b/stats/proc-vmstat.rb new file mode 100644 index 000000000..279257cb6 --- /dev/null +++ b/stats/proc-vmstat.rb @@ -0,0 +1,24 @@ +# input: each line of test log with Array(String) structure which like: +# [ +# "time: 1600586885", +# "nr_free_pages 87512", +# ... +# "time: 1600586886", +# "nr_free_pages 87787", +# ... +# ] +# return eg: +# { +# "time"=>[1600586885, 1600586886], +# "nr_free_pages"=>[87512, 87787], +# ... +# } +def proc_vmstat(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value.to_i + end + + result +end -- 2.23.0

1 0