- Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] sbin/makepkg: download submodules code from LKP server
by Liu Shaofei 14 Nov '20

14 Nov '20

Some repositories have submodules. So when build packages, we need download submodules from LKP server, avoid the waste of time in downloading files from the remote end. Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- sbin/makepkg | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sbin/makepkg b/sbin/makepkg index fde82212..c8b2ed00 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -623,6 +623,11 @@ extract_git() { cd_safe "${dir##*/}" + # modify submodules address to local server address. + if [[ -f ".gitmodules" ]] && [[ -n ${LKP_SERVER} ]]; then + echo -e "[url \"git://${LKP_SERVER}/\"]\n\tinsteadOf = https://" >> ~/.gitconfig + fi + local ref=origin/HEAD if [[ -n $fragment ]]; then case ${fragment%%=*} in -- 2.23.0

1 0

[PATCH v1 compass-ci] sched: change the result root dir
by Cao Xueliang 14 Nov '20

14 Nov '20

The new result root dir: /srv/result/$suite/$submit_date/$tbox_group/$rootfs/$pp_params/$job_id Example: rootfs: openeuler-20.09-aarch64 ($os-$os_version-$os_aarch) pp_params: socket-process-50 (use "-" connect pp's value up to 40 size) Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 18 +++++++++-- src/scheduler/pp_params.cr | 61 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 77 insertions(+), 2 deletions(-) create mode 100644 src/scheduler/pp_params.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index be0fdbb..4bef329 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -8,6 +8,7 @@ require "any_merge" require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" +require "scheduler/pp_params.cr" struct JSON::Any def []=(key : String, value : String) @@ -78,6 +79,9 @@ class Job initrd_pkg initrd_deps initrds_uri + rootfs + pp_params + submit_date result_root access_key access_key_file @@ -135,6 +139,9 @@ class Job append_init_field() set_user_lkp_src() set_os_dir() + set_submit_date() + set_pp_params() + set_rootfs() set_result_root() set_result_service() set_os_mount() @@ -188,10 +195,17 @@ class Job self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end + private def set_submit_date + self["submit_date"] = Time.local.to_s("%F") + end + + private def set_rootfs + self["rootfs"] = "#{os}-#{os_version}-#{os_arch}" + end + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group - date = Time.local.to_s("%F") - self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" + self["result_root"] = File.join("/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}", "#{pp_params}", "#{id}") # access_key has information based on "result_root" # so when set result_root, we need redo set_ to update it diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr new file mode 100644 index 0000000..28ec6ef --- /dev/null +++ b/src/scheduler/pp_params.cr @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + private def set_pp_params + temp_hash = Hash(String, JSON::Any).new + + if @hash["pp"] + flat_hash(@hash["pp"].as_h, temp_hash) + end + + self["pp_params"] = sort_keys_return_values(temp_hash) + end + + private def flat_hash(old_hash, new_hash) + old_hash.each do |key1, value1| + if value1.as_h? + next if value1.as_h.empty? + + temp_hash = Hash(String, JSON::Any).new + value1.as_h.each do |key2, value2| + temp_hash.merge!({"#{key1}-#{key2}" => value2}) + end + + flat_hash(temp_hash, new_hash) + else + new_hash.merge!({key1 => value1}) + end + end + end + + private def sort_keys_return_values(hash) + values = [] of String + items_size = 0 + + hash.keys.sort.each do |key| + value = format_string(hash[key].to_s) + next if 0 == value.size + + if 40 > value.size + items_size + values.size + values << value + items_size += value.size + else + break + end + end + + return values.join("-") + end + + private def format_string(original_str) + temp = [] of String + original_str.each_char do |char| + if "#{char}" =~ /-|\w/ + temp << "#{char}" + end + end + + return temp.join() + end +end -- 2.23.0

1 0

[PATCH compass-ci] sched: change the result root dir
by Cao Xueliang 14 Nov '20

14 Nov '20

The new result root dir: /srv/result/$suite/$submit_date/$tbox_group/$rootfs/$pp_params/$job_id Example: rootfs: openeuler-20.09-aarch64 ($os-$os_version-$os_aarch) pp_params: socket-process-50 (use "-" connect pp's value up to 40 size) Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 22 ++++++++++++-- src/scheduler/pp_params.cr | 61 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 81 insertions(+), 2 deletions(-) create mode 100644 src/scheduler/pp_params.cr diff --git a/src/lib/job.cr b/src/lib/job.cr index be0fdbb..f723100 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -8,6 +8,7 @@ require "any_merge" require "scheduler/constants.cr" require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" +require "scheduler/pp_params.cr" struct JSON::Any def []=(key : String, value : String) @@ -78,6 +79,9 @@ class Job initrd_pkg initrd_deps initrds_uri + rootfs + pp_params + submit_date result_root access_key access_key_file @@ -135,6 +139,9 @@ class Job append_init_field() set_user_lkp_src() set_os_dir() + set_submit_date() + set_pp_params() + set_rootfs() set_result_root() set_result_service() set_os_mount() @@ -188,10 +195,21 @@ class Job self["os_dir"] = "#{os}/#{os_arch}/#{os_version}" end + private def set_submit_date + self["submit_date"] = Time.local.to_s("%F") + end + + private def set_rootfs + self["rootfs"] = "#{os}-#{os_version}-#{os_arch}" + end + def set_result_root update_tbox_group_from_testbox # id must exists, need update tbox_group - date = Time.local.to_s("%F") - self["result_root"] = "/result/#{suite}/#{tbox_group}/#{date}/#{id}" + if pp_params != "" + self["result_root"] = "/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}/#{pp_params}/#{id}" + else + self["result_root"] = "/result/#{suite}/#{submit_date}/#{tbox_group}/#{rootfs}/#{id}" + end # access_key has information based on "result_root" # so when set result_root, we need redo set_ to update it diff --git a/src/scheduler/pp_params.cr b/src/scheduler/pp_params.cr new file mode 100644 index 0000000..28ec6ef --- /dev/null +++ b/src/scheduler/pp_params.cr @@ -0,0 +1,61 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +class Job + private def set_pp_params + temp_hash = Hash(String, JSON::Any).new + + if @hash["pp"] + flat_hash(@hash["pp"].as_h, temp_hash) + end + + self["pp_params"] = sort_keys_return_values(temp_hash) + end + + private def flat_hash(old_hash, new_hash) + old_hash.each do |key1, value1| + if value1.as_h? + next if value1.as_h.empty? + + temp_hash = Hash(String, JSON::Any).new + value1.as_h.each do |key2, value2| + temp_hash.merge!({"#{key1}-#{key2}" => value2}) + end + + flat_hash(temp_hash, new_hash) + else + new_hash.merge!({key1 => value1}) + end + end + end + + private def sort_keys_return_values(hash) + values = [] of String + items_size = 0 + + hash.keys.sort.each do |key| + value = format_string(hash[key].to_s) + next if 0 == value.size + + if 40 > value.size + items_size + values.size + values << value + items_size += value.size + else + break + end + end + + return values.join("-") + end + + private def format_string(original_str) + temp = [] of String + original_str.each_char do |char| + if "#{char}" =~ /-|\w/ + temp << "#{char}" + end + end + + return temp.join() + end +end -- 2.23.0

2 2

[PATCH compass-ci] container/kibana: fix build error
by Wu Zhende 14 Nov '20

14 Nov '20

Sending build context to Docker daemon 5.12kB Step 1/4 : ARG BASE_IMAGE Step 2/4 : FROM BASE_IMAGE invalid reference format: repository name must be lowercase Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 6e0dba0..970eb5a 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -3,7 +3,7 @@ ARG BASE_IMAGE -FROM BASE_IMAGE +FROM ${BASE_IMAGE} # docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/3] iso2rootfs: add some configurations after generated rootfs
by Yu Chuan 13 Nov '20

13 Nov '20

[Why] The initial rootfs have the follow two problems: - no dns resolver; - selinux is opened; Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 4f7843d7f348..05274164f8c1 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -338,6 +338,27 @@ run_qcow2rootfs() ./run "${qcow2_path}" "${ROOTFS_DES_DIR}" } + +############ config rootfs ############ +config_dns_resolver() +{ + cat <<-EOF > "${ROOTFS_DES_DIR}/etc/resolv.conf" + nameserver 114.114.114.114 + nameserver 8.8.8.8 + EOF +} + +disable_selinux() +{ + sed -i 's/^SELINUX=.*/SELINUX=disabled/g' "${ROOTFS_DES_DIR}/etc/selinux/config" +} + +config_rootfs() +{ + config_dns_resolver + disable_selinux +} + ############ test rootfs ############ get_qemu_efi_fd() { @@ -499,6 +520,8 @@ main() config_qcow2rootfs run_qcow2rootfs + config_rootfs + test_rootfs submit_test_job_yaml -- 2.23.0

2 2

[PATCH lkp-tests] tests/cci-makepkg: update gcc for centos7 when use initramfs os
by Lin Jiaxin 13 Nov '20

13 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/cci-makepkg | 15 ++++++++++++--- 1 file changed, 12 insertions(+), 3 deletions(-) diff --git a/tests/cci-makepkg b/tests/cci-makepkg index 67d055a4..bdb2128d 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -88,13 +88,22 @@ update_shared_pkg() echo "update shared pkg link ${benchmark}.cgz -> ${bm_name}/${cgz_name}" } +centos_update_gcc() +{ + if $(grep -sqF "CentOS Linux release 7" /etc/centos-release ); then + [[ "$os_mount" = "initramfs" ]] && { + yum install -y centos-release-scl-rh + yum install -y "devtoolset-8-gcc*" + } + source /opt/rh/devtoolset-8/enable + fi +} + [ -n "$PKGBUILD_TAG" ] && get_pkgfile distro_install_depends lkp-dev -[ $os = "centos" ] && -grep -sqF "CentOS Linux release 7" /etc/centos-release && -source /opt/rh/devtoolset-8/enable +[ $os = "centos" ] && centos_update_gcc distro_install_depends ${benchmark}-dev -- 2.23.0

1 0

[PATCH lkp-tests 2/2] depends: add mysql-dev
by Lin Jiaxin 13 Nov '20

13 Nov '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- distro/depends/mysql-dev | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 distro/depends/mysql-dev diff --git a/distro/depends/mysql-dev b/distro/depends/mysql-dev new file mode 100644 index 00000000..1434c26c --- /dev/null +++ b/distro/depends/mysql-dev @@ -0,0 +1,5 @@ +gcc +cmake +libssl-dev +libncurses5-dev +libtirpc-dev -- 2.23.0

1 0

[PATCH lkp-tests 1/2] pkg: add mysql
by Lin Jiaxin 13 Nov '20

13 Nov '20

sysbench-mysql need to test with mysql8.0.17 Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- ...00-mysql-add-fstack-protector-strong.patch | 20 +++++ pkg/mysql/PKGBUILD | 50 ++++++++++++ pkg/mysql/my.cnf | 80 +++++++++++++++++++ 3 files changed, 150 insertions(+) create mode 100644 pkg/mysql/0000-mysql-add-fstack-protector-strong.patch create mode 100644 pkg/mysql/PKGBUILD create mode 100644 pkg/mysql/my.cnf diff --git a/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch new file mode 100644 index 00000000..23c07c87 --- /dev/null +++ b/pkg/mysql/0000-mysql-add-fstack-protector-strong.patch @@ -0,0 +1,20 @@ +diff --git a/cmake/build_configurations/compiler_options.cmake b/cmake/build_configurations/compiler_options.cmake +index fecf2a2..f3e012f 100644 +--- a/cmake/build_configurations/compiler_options.cmake ++++ b/cmake/build_configurations/compiler_options.cmake +@@ -43,6 +43,7 @@ IF(UNIX) + # Default GCC flags + IF(CMAKE_COMPILER_IS_GNUCC) + SET(COMMON_C_FLAGS "-fno-omit-frame-pointer") ++ SET(COMMON_C_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_C_FLAGS "-fno-inline ") +@@ -57,6 +58,7 @@ IF(UNIX) + ENDIF() + IF(CMAKE_COMPILER_IS_GNUCXX) + SET(COMMON_CXX_FLAGS "-std=c++14 -fno-omit-frame-pointer") ++ SET(COMMON_CXX_FLAGS "-fstack-protector-strong") + # Disable inline optimizations for valgrind testing to avoid false positives + IF(WITH_VALGRIND) + STRING_PREPEND(COMMON_CXX_FLAGS "-fno-inline ") diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD new file mode 100644 index 00000000..f293f22f --- /dev/null +++ b/pkg/mysql/PKGBUILD @@ -0,0 +1,50 @@ +pkgname=mysql +pkgver=8.0.17 +pkgrel=3 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" + "0000-mysql-add-fstack-protector-strong.patch" + "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "my.cnf") +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'f1af6a77287e2456e867ecf66166679f') + +prepare() +{ + cd $srcdir/$pkgname-$pkgver + patch -p1 < ../0000-mysql-add-fstack-protector-strong.patch +} + +build() +{ + cd $srcdir/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + $CMAKE . -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc \ + -DWITH_INNOBASE_STORAGE_ENGINE=1 -DWITH_PARTITION_STORAGE_ENGINE=1 -DWITH_FEDERATED_STORAGE_ENGINE=1 \ + -DWITH_ARCHIVE_STORAGE_ENGINE=1 -DWITH_BLACKHOLE_STORAGE_ENGINE=1 -DWITH_MYISAM_STORAGE_ENGINE=1 \ + -DENABLED_LOCAL_INFILE=1 -DENABLE_DTRACE=0 -DDEFAULT_CHARSET=utf8mb4 -DDEFAULT_COLLATION=utf8mb4_general_ci \ + -DWITH_EMBEDDED_SERVER=1 -DDOWNLOAD_BOOST=1 -DWITH_BOOST=./boost -DFORCE_INSOURCE_BUILD=1 + make -j8 + + cd $srcdir + tar -zxvf getcpu.tgz + + cd getcpu || exit + make +} + +package() +{ + mkdir -p $pkgdir/getcpu + cp getcpu/getcpu-hi.ko $pkgdir/getcpu/ + + mkdir -p $pkgdir/etc + cp my.cnf $pkgdir/etc/ + + cd $srcdir/$pkgname-$pkgver + make DESTDIR=$pkgdir install +} diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf new file mode 100644 index 00000000..8d0b9c05 --- /dev/null +++ b/pkg/mysql/my.cnf @@ -0,0 +1,80 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +default-character-set=utf8mb4 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +default_authentication_plugin=mysql_native_password +port=3309 +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-grant-tables +#skip-name-resolve + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 +#transaction_isolation=READ_COMMITTED + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +#log-bin=mysql-bin +ssl=0 + +#innodb_doublewrite=0 +#innodb_flush_log_trx_commit=0 +#innodb_checksum_algorithm=none +#innodb_log_checksums=off +#foreign_key_checks=off +#unique_checks=off +table_open_cache=15000 + +#thread_handling=loaded-dynamically + + -- 2.23.0

1 0

[PATCH compass-ci] os-cifs/*: change base image name
by Wang Yong 13 Nov '20

13 Nov '20

update cifs base image name to maintain the consistency with os-http/os-nfs Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- container/os-cifs/build | 2 +- container/os-cifs/start | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/os-cifs/build b/container/os-cifs/build index 56ebbed..77be8e9 100755 --- a/container/os-cifs/build +++ b/container/os-cifs/build @@ -2,4 +2,4 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t alpine/samba . +docker build -t os-cifs . diff --git a/container/os-cifs/start b/container/os-cifs/start index 1b2cd23..f42c71c 100755 --- a/container/os-cifs/start +++ b/container/os-cifs/start @@ -19,7 +19,7 @@ cmd=( -v /srv/result:/srv/result --name os-cifs --restart=always - alpine/samba + os-cifs ) "${cmd[@]}" -- 2.23.0

1 0

[PATCH v7 compass-ci 11/11] mail-robot: documentation for service
by Luan Shengde 13 Nov '20

13 Nov '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/mail-robot.md | 97 +++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 97 insertions(+) create mode 100644 doc/mail-robot.md diff --git a/doc/mail-robot.md b/doc/mail-robot.md new file mode 100644 index 0000000..385ce96 --- /dev/null +++ b/doc/mail-robot.md @@ -0,0 +1,97 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ + +# email monitor robot + +## purpose + +apply account/uuid for user + uuid: used for user to submit jobs + account: used for user to check the test data + +## allowed email format: + +subject: apply account +mail content + commit url: + my oss commit: commit_url + example: + my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… +attahcment + ssh pub_key + file name: + - id_rsa.pub + +## steps overview + +1. mail-robot + monitor_new_email + - the monitor will listen to the mailbox for new email files + handle_new_email: + read email file content + apply_account + invoke AssignAccount for new account + +2. apply-account + init my_info + - my_email + - my_name + check_to_send_account + - parse_commit_url_pub_key + invoke ParseApplyAccountEmail + parse_commit_url + extract_commit_url + check whether there is a standard commit url + base_url_in_upstream_repos + check whether the repo's url in upstream-repos + commit_url_availability + check whether the commit url valid + - gitee_commit + clone the repo and check the commit + - non_gitee_commit(url) + check the commit with curl + + parse_pub_key + check whether there is a ssh pub_key in the email file + attachment: + first attachment + file_name: + id_ras.pub + + rescue error + error type: + - no commit url + - commit url not in upstream-repos + - commit url not available + - no pub_key + build_error_email with raised error message + send fail email + + - apply_my_account + my_uuid + generate uuid + apply_account + invoke ApplyJumperAccount to apply new account with my_info and pub_key + my_info: + - my_email + - my_name + - my_uuid + apply_jumper_account + required data: pub_key + complete my_info + my_info add: + - my_login_name + - my_commit_url + store my_info + invoke es to store the apply infos + + - send_mail + build_uuid_email + send mail + +3. continue the monitor + continue to monitor the mailbox for new email file + cycle run step 1 and 2 if matched email files + +4. service log + use the following command to get the logs + docker logs -f --tail=100 fluentd | grep mail-robot -- 2.23.0

1 0