April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sbin/download-rootfs: download osimage
by Liu Yinsi 23 Apr '21

23 Apr '21

the default os_mount=initramfs, so download osimage to run qemu kvm. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 38 ++++++++++++++++++++++++++------------ 1 file changed, 26 insertions(+), 12 deletions(-) diff --git a/sbin/download-rootfs b/sbin/download-rootfs index 68a68039..d936dbe6 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -19,22 +19,35 @@ os_version=${3:-20.03} load_cci_defaults rootfs_dir=/srv/os/${os}/${os_arch}/${os_version} +osimage_dir=/srv/initrd/osimage/${os}/${os_arch}/${os_version} -download_os_rootfs() -{ - mkdir -p $rootfs_dir - cd $rootfs_dir +mkdir -p $rootfs_dir $osimage_dir - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" - wget $download_url +download_one() +{ + local download_url=$1 + wget $download_url + + [ "$?" -ne 0 ] && { + log_error "failed to download os rootfs: $download_url" + exit 1 + } +} - [ "$?" -ne 0 ] && { - log_error "failed to download os rootfs: $download_url" - exit 1 - } +download_os_rootfs() +{ + cd $rootfs_dir + download_one "http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz +} - gzip -dc ${os_version}.cgz | cpio -id - rm -rf ${os_version}.cgz +download_osimage() +{ + cd $osimage_dir + download_one "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage/…" + download_one "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" } is_virtual() @@ -43,3 +56,4 @@ is_virtual() } is_virtual || download_os_rootfs +is_virtual || download_osimage -- 2.23.0

1 0

[PATCH v2 compass-ci] container/scheduler-http: fix unfriendly error
by Liu Yinsi 23 Apr '21

23 Apr '21

before: /c/compass-ci/container/scheduler-https# ./start Traceback (most recent call last): ./start:14:in `<main>': CA certificate is missing. Ignore if the HTTPS scheduler is not required (RuntimeError) after: /c/compass-ci/container/scheduler-https# ./start CA certificate is missing. Ignore if the HTTPS scheduler is not required Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/scheduler-https/start | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/container/scheduler-https/start b/container/scheduler-https/start index 4ec84a28..374300c0 100755 --- a/container/scheduler-https/start +++ b/container/scheduler-https/start @@ -11,7 +11,10 @@ crt_path = "#{ca_path}/server.crt" key_path = "#{ca_path}/server.key" warn_msg = 'CA certificate is missing. Ignore if the HTTPS scheduler is not required' -raise warn_msg unless File.exist?(crt_path) && File.exist?(key_path) +unless File.exist?(crt_path) && File.exist?(key_path) + puts warn_msg + exit +end names = Set.new %w[ SCHED_HOST -- 2.23.0

1 0

[PATCH compass-ci] container/es: support to build es images in x86_64 machine
by Liu Yinsi 23 Apr '21

23 Apr '21

choose base image according to different arch. arch: aarch64 => base image: elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786 arch: x86_64 => base image: elasticsearch:7.11.1 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/Dockerfile | 3 ++- container/es/build | 8 +++++++- 2 files changed, 9 insertions(+), 2 deletions(-) diff --git a/container/es/Dockerfile b/container/es/Dockerfile index f29e677e..bc320076 100644 --- a/container/es/Dockerfile +++ b/container/es/Dockerfile @@ -1,7 +1,8 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786 +ARG BASE_IMAGE +FROM ${BASE_IMAGE} ARG MEMORY ARG USER diff --git a/container/es/build b/container/es/build index e0232270..73593f4b 100755 --- a/container/es/build +++ b/container/es/build @@ -14,6 +14,12 @@ names = Set.new %w[ defaults = relevant_service_authentication(names) docker_skip_rebuild "es:7.11.1" +BASE_IMAGE_DICT = { + 'aarch64' => 'elasticsearch:7.11.1@sha256:d52cda1e73d1b1915ba2d76ca1e426620c7b5d6942d9d2f432259503974ba786', + 'x86_64' => 'elasticsearch:7.11.1' }.freeze + +BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] + available_memory = get_available_memory -system "docker build -t es:7.11.1 --build-arg MEMORY=#{available_memory} --build-arg USER=#{defaults['ES_USER']} --build-arg PASSWORD=#{defaults['ES_PASSWORD']} --network=host ." +system "docker build -t es:7.11.1 --build-arg MEMORY=#{available_memory} --build-arg USER=#{defaults['ES_USER']} --build-arg PASSWORD=#{defaults['ES_PASSWORD']} --build-arg BASE_IMAGE=#{BASE_IMAGE} --network=host ." -- 2.23.0

1 0

[PATCH compass-ci] container/kibana: configuring ES authentication information
by Wu Zhende 23 Apr '21

23 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/Dockerfile | 7 +++++-- container/kibana/build | 10 +++++++++- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/container/kibana/Dockerfile b/container/kibana/Dockerfile index 970eb5a..7085618 100644 --- a/container/kibana/Dockerfile +++ b/container/kibana/Dockerfile @@ -5,8 +5,11 @@ ARG BASE_IMAGE FROM ${BASE_IMAGE} -# docker image borrowed from hub.docker.com/r/gagara/kibana-oss-arm64 +ARG USER +ARG PASSWORD MAINTAINER Wu Zhende <wuzhende666(a)163.com> -RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml +RUN sed -i 's/server.host: "0"/server.host: "0.0.0.0"/' config/kibana.yml && \ + sed -i "6i\elasticsearch.username: ${USER}" config/kibana.yml && \ + sed -i "6i\elasticsearch.password: ${PASSWORD}" config/kibana.yml diff --git a/container/kibana/build b/container/kibana/build index 9c75aeb..c7bcd40 100755 --- a/container/kibana/build +++ b/container/kibana/build @@ -3,8 +3,16 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'set' require_relative '../defconfig' +names = Set.new %w[ + ES_USER + ES_PASSWORD +] + +defaults = relevant_service_authentication(names) + docker_skip_rebuild "kibana:7.11.1" BASE_IMAGE_DICT = { @@ -13,4 +21,4 @@ BASE_IMAGE_DICT = { BASE_IMAGE = BASE_IMAGE_DICT[%x(arch).chomp] -system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} ." +system "docker build -t kibana:7.11.1 --build-arg BASE_IMAGE=#{BASE_IMAGE} --build-arg USER=#{defaults['ES_USER']} --build-arg PASSWORD=#{defaults['ES_PASSWORD']} ." -- 2.23.0

1 0

[PATCH v2 compass-ci] container/etcd: adapt to the x86_64
by Cao Xueliang 23 Apr '21

23 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/Dockerfile | 9 +++++---- container/etcd/build | 11 ++++++++--- 2 files changed, 13 insertions(+), 7 deletions(-) diff --git a/container/etcd/Dockerfile b/container/etcd/Dockerfile index f4746ab..1612dbd 100644 --- a/container/etcd/Dockerfile +++ b/container/etcd/Dockerfile @@ -6,12 +6,13 @@ FROM alpine:3.13 MAINTAINER Cao Xueliang <caoxl78320(a)163.com> ARG VERSION +ARG ARCH RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && apk add ca-certificates && \ - wget https://repo.huaweicloud.com/etcd/${VERSION}/etcd-${VERSION}-linux-arm64.ta… && \ - tar zxvf etcd-${VERSION}-linux-arm64.tar.gz && \ - mv etcd-${VERSION}-linux-arm64/etcd* /bin/ && \ - rm -Rf etcd-${VERSION}-linux-arm64* /var/cache/apk/* + wget https://repo.huaweicloud.com/etcd/${VERSION}/etcd-${VERSION}-linux-${ARCH}.… && \ + tar zxvf etcd-${VERSION}-linux-${ARCH}.tar.gz && \ + mv etcd-${VERSION}-linux-${ARCH}/etcd* /bin/ && \ + rm -Rf etcd-${VERSION}-linux-${ARCH}* /var/cache/apk/* VOLUME /data diff --git a/container/etcd/build b/container/etcd/build index 37c2326..58def3d 100755 --- a/container/etcd/build +++ b/container/etcd/build @@ -4,7 +4,12 @@ . ../defconfig.sh -version=v3.4.15 -docker_skip_rebuild "etcd-${version}:alpine" +VERSION=v3.4.15 +docker_skip_rebuild "etcd-${VERSION}:alpine" -docker build -t etcd-${version}:alpine --build-arg VERSION=${version} . +[ "$(arch)" == "aarch64" ] && ARCH=arm64 +[ "$(arch)" == "x86_64" ] && ARCH=amd64 + +docker_skip_rebuild "etcd-${VERSION}:alpine" + +docker build -t etcd-${VERSION}:alpine --build-arg VERSION=${VERSION} --build-arg ARCH=${ARCH} . -- 2.23.0

1 0

[PATCH v2 lkp-tests] jobs: add keys for jobs
by Luan Shengde 23 Apr '21

23 Apr '21

add key for specify mysql socket Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/mysql-8-0-20.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/jobs/mysql-8-0-20.yaml b/jobs/mysql-8-0-20.yaml index 1926c5547..8c30f963e 100644 --- a/jobs/mysql-8-0-20.yaml +++ b/jobs/mysql-8-0-20.yaml @@ -32,3 +32,4 @@ sysbench-mysql: oltp_read_only: false report_interval: 1 events: 20000 + mysql_socket: /data/mysql/run/mysql.sock -- 2.23.0

1 0

[PATCH lkp-tests 5/6] tests: fix bugs
by Luan Shengde 23 Apr '21

23 Apr '21

del delete line: skip-grant-tables it will lead to access denied error. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 1 - 1 file changed, 1 deletion(-) diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 index ebc7c0e90..0c60ed9dd 100755 --- a/tests/mysql-8-0-20 +++ b/tests/mysql-8-0-20 @@ -40,7 +40,6 @@ configure_mysql() sleep 10 "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" - sed -i '/^skip-grant-tables/d' /etc/my.cnf "$mysql_server" restart sleep 10 } -- 2.23.0

2 2

[PATCH lkp-tests 4/6] depends: add depends list file for installing mysql
by Luan Shengde 23 Apr '21

23 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- distro/depends/mysql-8-0-20-dev | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 distro/depends/mysql-8-0-20-dev diff --git a/distro/depends/mysql-8-0-20-dev b/distro/depends/mysql-8-0-20-dev new file mode 100644 index 000000000..23c6ff661 --- /dev/null +++ b/distro/depends/mysql-8-0-20-dev @@ -0,0 +1,16 @@ +libnuma1 +xfsprogs +ethtool +gcc +cmake +libssl-dev +libncurses5-dev +libtirpc-dev +rpcsvc-proto-devel +rpcgen +gmp +gmp-devel +libmpc +libmpc-devel +mpfr +mpfr-devel -- 2.23.0

1 1

[PATCH lkp-tests 3/6] tests: add initialize file for mysql
by Luan Shengde 23 Apr '21

23 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 73 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 tests/mysql-8-0-20 diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 new file mode 100755 index 000000000..ebc7c0e90 --- /dev/null +++ b/tests/mysql-8-0-20 @@ -0,0 +1,73 @@ +#!/bin/sh +# - data_dev +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=127.0.0.1}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data + + mkdir -p /data/mysql/{data,share,tmp,run,log} + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 1

[PATCH lkp-tests 2/6] add pkgbuild file
by Luan Shengde 23 Apr '21

23 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- pkg/mysql-8-0-20/PKGBUILD | 79 +++++++++++++++++++++++++++++++++++++++ pkg/mysql-8-0-20/my.cnf | 69 ++++++++++++++++++++++++++++++++++ 2 files changed, 148 insertions(+) create mode 100644 pkg/mysql-8-0-20/PKGBUILD create mode 100644 pkg/mysql-8-0-20/my.cnf diff --git a/pkg/mysql-8-0-20/PKGBUILD b/pkg/mysql-8-0-20/PKGBUILD new file mode 100644 index 000000000..154f936cf --- /dev/null +++ b/pkg/mysql-8-0-20/PKGBUILD @@ -0,0 +1,79 @@ +pkgname=mysql-server-mysql +pkgver=8.0.20 +gccdir=gcc +pkgrel=1 +arch=('aarch64' 'x86_64') +url="http://www.mysql.com" +license=('GPLv2') +source=("https://gitee.com/li-yancheng/my-sql_with_boostkit_patch.git" + "https://gitee.com/li-yancheng/gcc.git" + "my.cnf") +md5sums=('SKIP' 'SKIP' '99c4965aa870c475000993f52a0dee66') + +build() +{ + # build_gcc + build_mysql +} + +build_gcc() +{ + mkdir -p $HOME/rpmbuild/SOURCES/ && { + cd $srcdir/gcc && cp -r * $HOME/rpmbuild/SOURCES/ + } + + cd $HOME/rpmbuild/SOURCES/ && { + rpmbuild -bp gcc.spec + } + + cd $HOME/rpmbuild/BUILD/gcc-9.3.0 && { + ./configure --prefix=/usr/ \ + --enable-shared \ + --enable-threads=posix \ + --enable-checking=release \ + --with-system-zlib \ + --enable-__cxa_atexit \ + --disable-libunwind-exceptions \ + --enable-gnu-unique-object \ + --enable-linker-build-id \ + --with-linker-hash-style=gnu \ + --enable-languages=c,c++,objc,obj-c++,fortran,lto \ + --enable-plugin \ + --enable-initfini-array \ + --disable-libgcj \ + --without-isl \ + --without-cloog \ + --enable-gnu-indirect-function \ + --with-stage1-ldflags='-Wl,-z,relro,-z,now' \ + --with-boot-ldflags='-Wl,-z,relro,-z,now' \ + --with-multilib-list=lp64 + make -j96 + make install + } +} + +build_mysql() +{ + unset CPPFLAGS + unset CFLAGS + unset CXXFLAGS + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver + + CMAKE=cmake + grep -sqF "CentOS Linux release 7" /etc/centos-release && CMAKE=cmake3 + + mkdir -p build + cd build && { + $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ + make -j96 + } +} + +package() +{ + cp my.cnf /etc/ + + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver/build && { + make install + } +} diff --git a/pkg/mysql-8-0-20/my.cnf b/pkg/mysql-8-0-20/my.cnf new file mode 100644 index 000000000..49e5bc3df --- /dev/null +++ b/pkg/mysql-8-0-20/my.cnf @@ -0,0 +1,69 @@ +[mysqld_safe] +log-error=/data/mysql/log/mysql.log +pid-file=/data/mysql/run/mysqld.pid +[mysqldump] +quick +[mysql] +[client] +# default-character-set=utf8mb4 +default-character-set=utf8 +socket=/data/mysql/run/mysql.sock + +[mysqld] +socket=/data/mysql/run/mysql.sock +tmpdir=/data/mysql/tmp +datadir=/data/mysql/data +basedir=/usr/local/mysql +default_authentication_plugin=mysql_native_password +user=mysql +memlock=on +bind-address=0.0.0.0 +skip-name-resolve +skip-grant-tables + +lc-messages=en_US +lc-messages-dir=/usr/local/mysql/share + +innodb_page_size=64k +max_connections=4000 +back_log=2048 +performance_schema=OFF +max_prepared_stmt_count=128000 + +#file +innodb_file_per_table +innodb_log_file_size=2048M +innodb_log_files_in_group=32 +innodb_open_files=10000 +table_open_cache_instances=64 + +#buffers +innodb_buffer_pool_size=128G +innodb_buffer_pool_instances=64 +innodb_log_buffer_size=2048M + +#tune +sync_binlog=0 +innodb_flush_log_at_trx_commit=0 +innodb_use_native_aio=1 +innodb_spin_wait_delay=160 +innodb_sync_spin_loops=20 +innodb_flush_method=O_DIRECT +innodb_io_capacity=30000 +innodb_io_capacity_max=40000 +innodb_lru_scan_depth=9000 +innodb_page_cleaners=1 +innodb_spin_wait_pause_multiplier=25 + +#perf special +innodb_flush_neighbors=0 +innodb_write_io_threads=1 +innodb_read_io_threads=1 +innodb_purge_threads=1 + +sql_mode=STRICT_TRANS_TABLES,NO_ENGINE_SUBSTITUTION,NO_AUTO_VALUE_ON_ZERO,STRICT_ALL_TABLES + +#skop_log_bin +ssl=0 + +table_open_cache=15000 -- 2.23.0

1 1