April 2021 - Compass-ci - mailweb.openeuler.org

help star
by Wu Fengguang 22 Apr '21

22 Apr '21

Yu Chuan is our help star, thanks Yu Chuan! 12.3 Yu Chuan 10.6 Wu Fengguang 9.8 Wei Jihui 9.2 Wang Yong 8.2 Liu Yinsi 7.1 Lin Jiaxin 6.7 Cao Xueliang 6.1 Liu Shaofei 5.6 Lu Weitao 3.8 Zhang Dewan 3.7 Wang Chenglong 3.7 Li Ping 3.2 Xu Xijian 2.5 Cui Yili 2.1 Luan Shengde 1.8 Bai Jing 1.8 Wu Zhende 1.8 Li Yuanchao 0.0 Du Kaitian 0.0 Zhang Yale

1 0

review star
by Wu Fengguang 22 Apr '21

22 Apr '21

Cao Xueliang is our review star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./review-star Review email records in the last 7 days total review paragraph point people 111 6 6 19 Cao Xueliang 20 4 7 5 Wei Jihui 15 3 5 4 Liu Shaofei 11 1 2 3 Yu Chuan 24 4 10 3 Luan Shengde 6 1 1 1 Lin Jiaxin 3 2 3 1 Xu Xijian 3 1 1 0 Wang Yong 1 0 0 0 Zhang Dewan 19 0 0 0 Liu Yinsi 6 1 2 0 Lu Weitao 21 1 1 0 Wu Zhende 18 1 2 0 Li Yuanchao 4 0 0 0 Wu Fengguang 15 0 0 0 Cui Yili 11 0 0 0 Wang Chenglong 11 0 0 0 Bai Jing 5 0 0 0 Li Ping

1 0

git count
by Wu Fengguang 22 Apr '21

22 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 16 Cao Xueliang 15 Wu Zhende 10 Li Yuanchao 6 Liu Yinsi 6 Yu Chuan 5 Liu Shaofei 5 Luan Shengde 4 Bai Jing 3 Wang Chenglong 2 Lu Weitao 2 Wang Yong 1 Xu Xijian 1 Lin Jiaxin 1 Li Ping 0 Du Kaitian 0 Wei Jihui 0 Wu Fengguang 0 Zhang Yale 0 Cui Yili 0 Zhang Dewan

1 0

[PATCH v2 compass-ci] es_query.rb: fix _scroll_id change
by Cui Yili 22 Apr '21

22 Apr '21

[why] _scroll_id will change every time query, so you need to save the last _scroll_id. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- lib/es_query.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index dd6d97dc..8c5a28fe 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -45,11 +45,12 @@ class ESQuery }, size: size } result = @client.search index: @index, scroll: '10m', body: query - @scroll_id = result['_scroll_id'] - return result else - @client.scroll scroll: '10m', scroll_id: @scroll_id + result = @client.scroll scroll: '10m', scroll_id: @scroll_id end + + @scroll_id = result['_scroll_id'] unless result.empty? && result.include?('_scroll_id') + return result end def query_by_id(id) -- 2.23.0

1 0

[PATCH compass-ci] dracut-initrd: support local disk rootfs
by Yu Chuan 22 Apr '21

22 Apr '21

[Why] Allow user to submit a job that rootfs is in local disk by add 'custom_ipxe' field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 9e8374290230..1a2edafaf967 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -2,6 +2,10 @@ analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" + + # if root is a local disk, then boot directly. + [[ $rootfs =~ ^/dev/ ]] && exit 0 + # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') timestamp="$(basename $rootfs_src)" -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/bootstrap.sh: delete history remove code
by Liu Shaofei 22 Apr '21

22 Apr '21

[why] function callend by "is_clearlinux" is not defined, by checking history commit record, the function is removed and the patch codes is removed. [ 63.427966] [ 63.428704] + is_clearlinux [ 63.428705] [ 63.431226] /lkp/lkp/src/lib/bootstrap.sh: line 104: is_clearlinux: command not found [ 63.431228] Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- lib/bootstrap.sh | 7 ------- 1 file changed, 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7983f638e..070dabf46 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -101,13 +101,6 @@ network_ok() [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && test_ping && return 0 done - is_clearlinux && { - net_devices_link up - - # in case of systemd-networkd.service was masked - systemctl unmask systemd-networkd.service - systemctl start systemd-networkd.service - } return 1 } -- 2.23.0

2 1

[PATCH compass-ci] services: optimization logs
by Wu Zhende 22 Apr '21

22 Apr '21

1. use "e.inspect_with_backtrace" to print more detailed error information 2. add data to save data when an error is reported Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 58 ++++++++++++++------------ src/lib/mq.cr | 2 +- src/lib/sched.cr | 14 +++---- src/monitoring/amqp.cr | 12 +++++- src/monitoring/filter.cr | 6 ++- src/scheduler/close_job.cr | 2 +- src/scheduler/download_file.cr | 2 +- src/scheduler/find_next_job_boot.cr | 2 +- src/scheduler/request_cluster_state.cr | 2 +- 9 files changed, 59 insertions(+), 41 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index e56a681..fe870de 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -45,7 +45,7 @@ class Lifecycle rescue e @log.warn({ "resource" => "/alive", - "message" => e.to_s + "message" => e.inspect_with_backtrace }.to_json) end @@ -74,7 +74,7 @@ class Lifecycle rescue e @log.warn({ "resource" => "init_from_es", - "message" => e.to_s + "message" => e.inspect_with_backtrace }.to_json) end @@ -118,30 +118,34 @@ class Lifecycle def mq_event_loop puts "deal job events" - q = @mq.ch.queue("job_mq", durable: false) - q.subscribe(no_ack: false) do |msg| - event = JSON.parse(msg.body_io.to_s) - job_state = event["job_state"]? - - case job_state - when "boot" - on_job_boot(event) - when "close" - on_job_close(event) - when "abnormal" - on_abnormal_job(event) - when "crash" - on_job_crash(event) - else - on_other_job(event) + event = JSON::Any.new(nil) + begin + q = @mq.ch.queue("job_mq", durable: false) + q.subscribe(no_ack: false) do |msg| + event = JSON.parse(msg.body_io.to_s) + job_state = event["job_state"]? + + case job_state + when "boot" + on_job_boot(event) + when "close" + on_job_close(event) + when "abnormal" + on_abnormal_job(event) + when "crash" + on_job_crash(event) + else + on_other_job(event) + end + @mq.ch.basic_ack(msg.delivery_tag) end - @mq.ch.basic_ack(msg.delivery_tag) + rescue e + @log.warn({ + "resource" => "mq_event_loop", + "message" => e.inspect_with_backtrace, + "event" => event + }.to_json) end - rescue e - @log.warn({ - "resource" => "mq_event_loop", - "message" => e.to_s - }.to_json) end def on_other_job(event) @@ -269,7 +273,8 @@ class Lifecycle rescue e @log.warn({ "resource" => "timeout_job_loop", - "message" => e.to_s + "message" => e.inspect_with_backtrace, + "job_id" => dead_job_id }.to_json) end end @@ -285,7 +290,8 @@ class Lifecycle rescue e @log.warn({ "resource" => "timeout_machine_loop", - "message" => e.to_s + "message" => e.inspect_with_backtrace, + "testbox" => dead_machine_name }.to_json) end end diff --git a/src/lib/mq.cr b/src/lib/mq.cr index 1c7f4b7..67fcee2 100644 --- a/src/lib/mq.cr +++ b/src/lib/mq.cr @@ -28,7 +28,7 @@ class MQClient }.to_json) rescue e @log.warn({ - "msg" => e.to_s, + "msg" => e.inspect_with_backtrace, "source" => "mq_client" }.to_json) end diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 7a9e4d8..43fe47b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -56,7 +56,7 @@ class Sched def alive(version) "LKP Alive! The time is #{Time.local}, version = #{version}" rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def normalize_mac(mac : String) @@ -72,7 +72,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def del_host_mac @@ -84,7 +84,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def set_host2queues @@ -96,7 +96,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def del_host2queues @@ -108,7 +108,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def get_time @@ -143,7 +143,7 @@ class Sched hash["testbox"] = testbox @log.info(hash.to_json) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def send_mq_msg(job_state) @@ -223,7 +223,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/monitoring/amqp.cr b/src/monitoring/amqp.cr index f23113a..6ba7b1b 100644 --- a/src/monitoring/amqp.cr +++ b/src/monitoring/amqp.cr @@ -29,7 +29,11 @@ class MessageQueueClient begin filter.filter_msg(msg.body_io) rescue e - @log.warn("filter message failed: #{e}") + @log.warn({ + "resource" => "filter_message", + "message" => e.inspect_with_backtrace, + "data" => msg.body_io.to_s + }.to_json) end end end @@ -42,7 +46,11 @@ class MessageQueueClient filter_msg(conn, filter, exchange_name, queue_name) end rescue e - @log.warn("monitoring_message_queue failed: #{e}") + @log.warn({ + "resource" => "monitoring_message_queue", + "message" => e.inspect_with_backtrace, + "data" => "#{exchange_name}, #{queue_name}" + }.to_json) sleep 5 end end diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index b663f11..fe41571 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -47,7 +47,11 @@ class Filter @hash[query].each do |socket| socket.send msg.to_json rescue e - @log.warn("send msg failed: #{e}") + @log.warn({ + "resource" => "send_msg", + "message" => e.inspect_with_backtrace, + "data" => "query: #{query}, mas: #{msg}" + }.to_json) remove_filter_rule(query, socket) end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index a112a38..0f051e6 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -30,7 +30,7 @@ class Sched job_state ||= "complete" @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) ensure source = @env.params.query["source"]? if source != "lifecycle" diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr index 50e27ba..48ce55a 100644 --- a/src/scheduler/download_file.cr +++ b/src/scheduler/download_file.cr @@ -11,6 +11,6 @@ class Sched send_file @env, file_path rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end end diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 091c41a..15daebf 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -15,6 +15,6 @@ class Sched response rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index e8ef6f3..e9be3cf 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -69,7 +69,7 @@ class Sched # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end # node_state: "finish" | "ready" -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: add function set_os_arch()
by Liu Yinsi 22 Apr '21

22 Apr '21

before: submit host-info.yaml os_arch=x86_64 arch=x86_64 after: submit host-info.yaml arch=x86_64 [why] when submit job to docker testbox in x86_64 machine, must use params both os_arch and arch, or failed to submit and consume job, now just assign value to arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- src/lib/job.cr | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 200ba2b1..bb4ef967 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -151,6 +151,7 @@ class Job private def set_defaults append_init_field() + set_os_arch() set_docker_os() set_user_lkp_src() set_os_mount() @@ -174,6 +175,10 @@ class Job self[key] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") end + private def set_os_arch + self["os_arch"] = @hash["arch"].to_s if @hash.has_key?("arch") + end + private def set_docker_os return unless is_docker_job? -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/4] depends: add depends list file for installing mysql
by Luan Shengde 22 Apr '21

22 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- distro/depends/mysql-8-0-20-dev | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 distro/depends/mysql-8-0-20-dev diff --git a/distro/depends/mysql-8-0-20-dev b/distro/depends/mysql-8-0-20-dev new file mode 100644 index 000000000..23c6ff661 --- /dev/null +++ b/distro/depends/mysql-8-0-20-dev @@ -0,0 +1,16 @@ +libnuma1 +xfsprogs +ethtool +gcc +cmake +libssl-dev +libncurses5-dev +libtirpc-dev +rpcsvc-proto-devel +rpcgen +gmp +gmp-devel +libmpc +libmpc-devel +mpfr +mpfr-devel -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/4] tests: add initialize file for mysql
by Luan Shengde 22 Apr '21

22 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 tests/mysql-8-0-20 diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 new file mode 100755 index 000000000..ebc7c0e90 --- /dev/null +++ b/tests/mysql-8-0-20 @@ -0,0 +1,73 @@ +#!/bin/sh +# - data_dev +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=127.0.0.1}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data + + mkdir -p /data/mysql/{data,share,tmp,run,log} + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 0