- Compass-ci - mailweb.openeuler.org

git count
by Wu Fengguang 22 Apr '21

22 Apr '21

Cao Xueliang is our code star, thanks Cao Xueliang! baijing@crystal ~/todo/bin% ./git-count 16 Cao Xueliang 15 Wu Zhende 10 Li Yuanchao 6 Liu Yinsi 6 Yu Chuan 5 Liu Shaofei 5 Luan Shengde 4 Bai Jing 3 Wang Chenglong 2 Lu Weitao 2 Wang Yong 1 Xu Xijian 1 Lin Jiaxin 1 Li Ping 0 Du Kaitian 0 Wei Jihui 0 Wu Fengguang 0 Zhang Yale 0 Cui Yili 0 Zhang Dewan

1 0

[PATCH v2 compass-ci] es_query.rb: fix _scroll_id change
by Cui Yili 22 Apr '21

22 Apr '21

[why] _scroll_id will change every time query, so you need to save the last _scroll_id. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- lib/es_query.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index dd6d97dc..8c5a28fe 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -45,11 +45,12 @@ class ESQuery }, size: size } result = @client.search index: @index, scroll: '10m', body: query - @scroll_id = result['_scroll_id'] - return result else - @client.scroll scroll: '10m', scroll_id: @scroll_id + result = @client.scroll scroll: '10m', scroll_id: @scroll_id end + + @scroll_id = result['_scroll_id'] unless result.empty? && result.include?('_scroll_id') + return result end def query_by_id(id) -- 2.23.0

1 0

[PATCH compass-ci] dracut-initrd: support local disk rootfs
by Yu Chuan 22 Apr '21

22 Apr '21

[Why] Allow user to submit a job that rootfs is in local disk by add 'custom_ipxe' field in job.yaml. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- container/dracut-initrd/bin/set-local-sysroot.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/container/dracut-initrd/bin/set-local-sysroot.sh b/container/dracut-initrd/bin/set-local-sysroot.sh index 9e8374290230..1a2edafaf967 100644 --- a/container/dracut-initrd/bin/set-local-sysroot.sh +++ b/container/dracut-initrd/bin/set-local-sysroot.sh @@ -2,6 +2,10 @@ analyse_kernel_cmdline_params() { rootfs="$(getarg root=)" + + # if root is a local disk, then boot directly. + [[ $rootfs =~ ^/dev/ ]] && exit 0 + # example: $nfs_server_ip:/os/${os}/${os_arch}/${os_version}-snapshots/20210310005959 rootfs_src=$(echo $"$rootfs" | sed 's/\///') timestamp="$(basename $rootfs_src)" -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/bootstrap.sh: delete history remove code
by Liu Shaofei 22 Apr '21

22 Apr '21

[why] function callend by "is_clearlinux" is not defined, by checking history commit record, the function is removed and the patch codes is removed. [ 63.427966] [ 63.428704] + is_clearlinux [ 63.428705] [ 63.431226] /lkp/lkp/src/lib/bootstrap.sh: line 104: is_clearlinux: command not found [ 63.431228] Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- lib/bootstrap.sh | 7 ------- 1 file changed, 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7983f638e..070dabf46 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -101,13 +101,6 @@ network_ok() [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && test_ping && return 0 done - is_clearlinux && { - net_devices_link up - - # in case of systemd-networkd.service was masked - systemctl unmask systemd-networkd.service - systemctl start systemd-networkd.service - } return 1 } -- 2.23.0

2 1

[PATCH compass-ci] services: optimization logs
by Wu Zhende 22 Apr '21

22 Apr '21

1. use "e.inspect_with_backtrace" to print more detailed error information 2. add data to save data when an error is reported Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 58 ++++++++++++++------------ src/lib/mq.cr | 2 +- src/lib/sched.cr | 14 +++---- src/monitoring/amqp.cr | 12 +++++- src/monitoring/filter.cr | 6 ++- src/scheduler/close_job.cr | 2 +- src/scheduler/download_file.cr | 2 +- src/scheduler/find_next_job_boot.cr | 2 +- src/scheduler/request_cluster_state.cr | 2 +- 9 files changed, 59 insertions(+), 41 deletions(-) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index e56a681..fe870de 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -45,7 +45,7 @@ class Lifecycle rescue e @log.warn({ "resource" => "/alive", - "message" => e.to_s + "message" => e.inspect_with_backtrace }.to_json) end @@ -74,7 +74,7 @@ class Lifecycle rescue e @log.warn({ "resource" => "init_from_es", - "message" => e.to_s + "message" => e.inspect_with_backtrace }.to_json) end @@ -118,30 +118,34 @@ class Lifecycle def mq_event_loop puts "deal job events" - q = @mq.ch.queue("job_mq", durable: false) - q.subscribe(no_ack: false) do |msg| - event = JSON.parse(msg.body_io.to_s) - job_state = event["job_state"]? - - case job_state - when "boot" - on_job_boot(event) - when "close" - on_job_close(event) - when "abnormal" - on_abnormal_job(event) - when "crash" - on_job_crash(event) - else - on_other_job(event) + event = JSON::Any.new(nil) + begin + q = @mq.ch.queue("job_mq", durable: false) + q.subscribe(no_ack: false) do |msg| + event = JSON.parse(msg.body_io.to_s) + job_state = event["job_state"]? + + case job_state + when "boot" + on_job_boot(event) + when "close" + on_job_close(event) + when "abnormal" + on_abnormal_job(event) + when "crash" + on_job_crash(event) + else + on_other_job(event) + end + @mq.ch.basic_ack(msg.delivery_tag) end - @mq.ch.basic_ack(msg.delivery_tag) + rescue e + @log.warn({ + "resource" => "mq_event_loop", + "message" => e.inspect_with_backtrace, + "event" => event + }.to_json) end - rescue e - @log.warn({ - "resource" => "mq_event_loop", - "message" => e.to_s - }.to_json) end def on_other_job(event) @@ -269,7 +273,8 @@ class Lifecycle rescue e @log.warn({ "resource" => "timeout_job_loop", - "message" => e.to_s + "message" => e.inspect_with_backtrace, + "job_id" => dead_job_id }.to_json) end end @@ -285,7 +290,8 @@ class Lifecycle rescue e @log.warn({ "resource" => "timeout_machine_loop", - "message" => e.to_s + "message" => e.inspect_with_backtrace, + "testbox" => dead_machine_name }.to_json) end end diff --git a/src/lib/mq.cr b/src/lib/mq.cr index 1c7f4b7..67fcee2 100644 --- a/src/lib/mq.cr +++ b/src/lib/mq.cr @@ -28,7 +28,7 @@ class MQClient }.to_json) rescue e @log.warn({ - "msg" => e.to_s, + "msg" => e.inspect_with_backtrace, "source" => "mq_client" }.to_json) end diff --git a/src/lib/sched.cr b/src/lib/sched.cr index 7a9e4d8..43fe47b 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -56,7 +56,7 @@ class Sched def alive(version) "LKP Alive! The time is #{Time.local}, version = #{version}" rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def normalize_mac(mac : String) @@ -72,7 +72,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def del_host_mac @@ -84,7 +84,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def set_host2queues @@ -96,7 +96,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def del_host2queues @@ -108,7 +108,7 @@ class Sched "No yet!" end rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def get_time @@ -143,7 +143,7 @@ class Sched hash["testbox"] = testbox @log.info(hash.to_json) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end def send_mq_msg(job_state) @@ -223,7 +223,7 @@ class Sched @log.info(%({"job_id": "#{job_id}", "state": "set ssh port", "ssh_port": "#{ssh_port}", "tbox_name": "#{testbox}"})) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end private def query_consumable_keys(shortest_queue_name) diff --git a/src/monitoring/amqp.cr b/src/monitoring/amqp.cr index f23113a..6ba7b1b 100644 --- a/src/monitoring/amqp.cr +++ b/src/monitoring/amqp.cr @@ -29,7 +29,11 @@ class MessageQueueClient begin filter.filter_msg(msg.body_io) rescue e - @log.warn("filter message failed: #{e}") + @log.warn({ + "resource" => "filter_message", + "message" => e.inspect_with_backtrace, + "data" => msg.body_io.to_s + }.to_json) end end end @@ -42,7 +46,11 @@ class MessageQueueClient filter_msg(conn, filter, exchange_name, queue_name) end rescue e - @log.warn("monitoring_message_queue failed: #{e}") + @log.warn({ + "resource" => "monitoring_message_queue", + "message" => e.inspect_with_backtrace, + "data" => "#{exchange_name}, #{queue_name}" + }.to_json) sleep 5 end end diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index b663f11..fe41571 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -47,7 +47,11 @@ class Filter @hash[query].each do |socket| socket.send msg.to_json rescue e - @log.warn("send msg failed: #{e}") + @log.warn({ + "resource" => "send_msg", + "message" => e.inspect_with_backtrace, + "data" => "query: #{query}, mas: #{msg}" + }.to_json) remove_filter_rule(query, socket) end end diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index a112a38..0f051e6 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -30,7 +30,7 @@ class Sched job_state ||= "complete" @log.info(%({"job_id": "#{job_id}", "job_state": "#{job_state}"})) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) ensure source = @env.params.query["source"]? if source != "lifecycle" diff --git a/src/scheduler/download_file.cr b/src/scheduler/download_file.cr index 50e27ba..48ce55a 100644 --- a/src/scheduler/download_file.cr +++ b/src/scheduler/download_file.cr @@ -11,6 +11,6 @@ class Sched send_file @env, file_path rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end end diff --git a/src/scheduler/find_next_job_boot.cr b/src/scheduler/find_next_job_boot.cr index 091c41a..15daebf 100644 --- a/src/scheduler/find_next_job_boot.cr +++ b/src/scheduler/find_next_job_boot.cr @@ -15,6 +15,6 @@ class Sched response rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end end diff --git a/src/scheduler/request_cluster_state.cr b/src/scheduler/request_cluster_state.cr index e8ef6f3..e9be3cf 100644 --- a/src/scheduler/request_cluster_state.cr +++ b/src/scheduler/request_cluster_state.cr @@ -69,7 +69,7 @@ class Sched # show cluster state return @redis.hash_get("sched/cluster_state", cluster_id) rescue e - @log.warn(e) + @log.warn(e.inspect_with_backtrace) end # node_state: "finish" | "ready" -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: add function set_os_arch()
by Liu Yinsi 22 Apr '21

22 Apr '21

before: submit host-info.yaml os_arch=x86_64 arch=x86_64 after: submit host-info.yaml arch=x86_64 [why] when submit job to docker testbox in x86_64 machine, must use params both os_arch and arch, or failed to submit and consume job, now just assign value to arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- src/lib/job.cr | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 200ba2b1..bb4ef967 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -151,6 +151,7 @@ class Job private def set_defaults append_init_field() + set_os_arch() set_docker_os() set_user_lkp_src() set_os_mount() @@ -174,6 +175,10 @@ class Job self[key] = Time.local.to_s("%Y-%m-%dT%H:%M:%S+0800") end + private def set_os_arch + self["os_arch"] = @hash["arch"].to_s if @hash.has_key?("arch") + end + private def set_docker_os return unless is_docker_job? -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/4] depends: add depends list file for installing mysql
by Luan Shengde 22 Apr '21

22 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- distro/depends/mysql-8-0-20-dev | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 distro/depends/mysql-8-0-20-dev diff --git a/distro/depends/mysql-8-0-20-dev b/distro/depends/mysql-8-0-20-dev new file mode 100644 index 000000000..23c6ff661 --- /dev/null +++ b/distro/depends/mysql-8-0-20-dev @@ -0,0 +1,16 @@ +libnuma1 +xfsprogs +ethtool +gcc +cmake +libssl-dev +libncurses5-dev +libtirpc-dev +rpcsvc-proto-devel +rpcgen +gmp +gmp-devel +libmpc +libmpc-devel +mpfr +mpfr-devel -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/4] tests: add initialize file for mysql
by Luan Shengde 22 Apr '21

22 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 tests/mysql-8-0-20 diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 new file mode 100755 index 000000000..ebc7c0e90 --- /dev/null +++ b/tests/mysql-8-0-20 @@ -0,0 +1,73 @@ +#!/bin/sh +# - data_dev +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=127.0.0.1}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data + + mkdir -p /data/mysql/{data,share,tmp,run,log} + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/4] add pkgbuild file
by Luan Shengde 22 Apr '21

22 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- pkg/mysql-8-0-20/PKGBUILD | 79 +++++++++++++++++++++++++++++++++++++++ pkg/mysql-8-0-20/my.cnf | 69 ++++++++++++++++++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 pkg/mysql-8-0-20/PKGBUILD create mode 100644 pkg/mysql-8-0-20/my.cnf diff --git a/pkg/mysql-8-0-20/PKGBUILD b/pkg/mysql-8-0-20/PKGBUILD new file mode 100644 index 000000000..154f936cf --- /dev/null +++ b/pkg/mysql-8-0-20/PKGBUILD @@ -0,0 +1,79 @@ +pkgname=mysql-server-mysql +pkgver=8.0.20 +gccdir=gcc +pkgrel=1 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://gitee.com/li-yancheng/my-sql_with_boostkit_patch.git" + "https://gitee.com/li-yancheng/gcc.git" + "my.cnf") +md5sums=('SKIP' 'SKIP' '99c4965aa870c475000993f52a0dee66') + +build() +{ + # build_gcc + build_mysql +} + +build_gcc() +{ + mkdir -p $HOME/rpmbuild/SOURCES/ && { + cd $srcdir/gcc && cp -r * $HOME/rpmbuild/SOURCES/ + } + + cd $HOME/rpmbuild/SOURCES/ && { + rpmbuild -bp gcc.spec + } + + cd $HOME/rpmbuild/BUILD/gcc-9.3.0 && { + ./configure --prefix=/usr/ \ + --enable-shared \ + --enable-threads=posix \ + --enable-checking=release \ + --with-system-zlib \ + --enable-__cxa_atexit \ + --disable-libunwind-exceptions \ + --enable-gnu-unique-object \ + --enable-linker-build-id \ + --with-linker-hash-style=gnu \ + --enable-languages=c,c++,objc,obj-c++,fortran,lto \ + --enable-plugin \ + --enable-initfini-array \ + --disable-libgcj \ + --without-isl \ + --without-cloog \ + --enable-gnu-indirect-function \ + --with-stage1-ldflags='-Wl,-z,relro,-z,now' \ + --with-boot-ldflags='-Wl,-z,relro,-z,now' \ + --with-multilib-list=lp64 + make -j96 + make install + } +} + +build_mysql() +{ + unset CPPFLAGS + unset CFLAGS + unset CXXFLAGS + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + mkdir -p build + cd build && { + $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ + make -j96 + } +} + +package() +{ + cp my.cnf /etc/ + + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver/build && { + make install + } +} diff --git a/pkg/mysql-8-0-20/my.cnf b/pkg/mysql-8-0-20/my.cnf new file mode 100644 index 000000000..49e5bc3df --- /dev/null +++ b/pkg/mysql-8-0-20/my.cnf @@ -0,0 +1,69 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +# default-character-set=utf8mb4 +default-character-set=utf8 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +basedir=/usr/local/mysql +default_authentication_plugin=mysql_native_password +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-name-resolve +skip-grant-tables + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +ssl=0 + +table_open_cache=15000 -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/4] jobs: add job file for installing/testing mysql
by Luan Shengde 22 Apr '21

22 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/mysql-8-0-20.yaml | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 jobs/mysql-8-0-20.yaml diff --git a/jobs/mysql-8-0-20.yaml b/jobs/mysql-8-0-20.yaml new file mode 100644 index 000000000..1926c5547 --- /dev/null +++ b/jobs/mysql-8-0-20.yaml @@ -0,0 +1,34 @@ +suite: cci-makepkg +category: functional + +testbox: taishan200-2280-2s48p-256g--a18 + +os: openeuler +os_version: 20.03 +os_mount: initramfs + +mysql_user: +mysql_host: +mysql_port: +mysql_db: +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 60 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: false + report_interval: 1 + events: 20000 -- 2.23.0

1 0