January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] pkg/mysql/PKGBUILD: fix mysql pkgbuild
by Wei Jihui 27 Jan '21

27 Jan '21

[error] CMake Error at plugin/group_replication/libmysqlgcs/rpcgen.cmake:100(MESSAGE): Could not find rpcgen ... -- Configuring incomplete, errors occurred! [how] make and make install rpcgen Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/mysql/PKGBUILD | 12 +++++++++++- 1 file changed, 11 insertions(+), 1 deletion(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 8869e62b..0f7d8bc8 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -7,8 +7,9 @@ license=('GPLv2') source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "0000-mysql-add-fstack-protector-strong.patch" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" + "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' '7bb0372ec4e9281813a9afc67c6cfc20') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '7bb0372ec4e9281813a9afc67c6cfc20') prepare() { @@ -19,9 +20,18 @@ prepare() build() { build_getcpu + build_rpcsvc build_mysql } +build_rpcsvc() +{ + cd "$srcdir/rpcsvc-proto-1.4" + ./configure + make + make install +} + build_mysql() { cd $srcdir/$pkgname-$pkgver -- 2.23.0

1 0

[PATCH compass-ci] lib/matrix.rb: fix NoMethodError
by Lu Weitao 27 Jan '21

27 Jan '21

[Why] /home/wangyong/compass-ci/lib/matrix2.rb:21:in `set_pre_value': undefined method `sum' for 1:Integer (NoMethodError) Did you mean? succ root cause: if a job has *.yaml(except job.yaml) in result_root, will meet this error when merge it to stats.json directly. [How] remove the logical handle *.yaml from lib/matrix.rb at first, we should only handle $script.yaml, and the feature will be done in other patch Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/matrix2.rb | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/lib/matrix2.rb b/lib/matrix2.rb index 8444334..4dd03b4 100644 --- a/lib/matrix2.rb +++ b/lib/matrix2.rb @@ -39,21 +39,19 @@ def file_check(file) File.basename(file, '.json') when /\.json\.gz$/ File.basename(file, '.json.gz') - when /\.yaml$/ - File.basename(file, '.yaml') end end def create_stats(result_root) stats = {} - monitor_files = Dir["#{result_root}/*.{json,json.gz,yaml}"] + monitor_files = Dir["#{result_root}/*.{json,json.gz}"] monitor_files.each do |file| next unless File.size?(file) monitor = file_check(file) - next if %w[stats job].include?(monitor) + next if monitor == 'stats' extract_pre_result(stats, monitor, file) end -- 2.23.0

1 0

[PATCH v2 lkp-tests] tests/build-pkg: export V=1 to print verbose output
by Lin Jiaxin 27 Jan '21

27 Jan '21

A developer who really wants to make silent rules enabled by default will calling AM_SILENT_RULES([yes]) in configure.ac. It make compiling packages by default does not show the full compilation/link command lines, but only short "summaries", like this: [before] EPOINFO main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/ctags-cmd.o make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' CC main/libctags_a-args.o However, we need the full compilation/link command lines, like this: [after] ./misc/gen-repoinfo main/repoinfo.h make all-am make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' gcc -DHAVE_CONFIG_H -I. -I. -I. -I./main -I./peg -DHAVE_PACKCC -DUSE_SYSTEM_STRNLEN -DHAVE_REPOINFO_H -D_FORTIFY_SOURCE=2 -std=gnu99 -Wall -I/usr/include/libxml2 -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -MT main/ctags-cmd.o -MD -MP -MF main/.deps/ctags-cmd.Tpo -c -o main/ctags-cmd.o `test -f 'main/cmd.c' || echo './'`main/cmd.c make[1]: Leaving directory '/tmp/lkp/uctags-git/src/ctags' make[1]: Entering directory '/tmp/lkp/uctags-git/src/ctags' mv -f main/.deps/ctags-cmd.Tpo main/.deps/ctags-cmd.Po make V=1 will produce verbose output, so export V=1 in build-pkg script. refer to: https://www.gnu.org/software/automake/manual/html_node/Automake-Silent-Rule… https://lists.gnu.org/archive/html/bug-autoconf/2012-01/msg00009.html Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 5344da2a..f3545a19 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -157,6 +157,8 @@ build_source_pkg() local mark="true" [ "${os}" == "archlinux" ] && local mark="pacman" + export V=1 + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf -- 2.23.0

1 0

[PATCH compass-ci] update email address to apply account in README
by Li Yuanchao 27 Jan '21

27 Jan '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- README.en.md | 2 +- README.zh.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/README.en.md b/README.en.md index 632a480..fa88305 100644 --- a/README.en.md +++ b/README.en.md @@ -44,7 +44,7 @@ If a new error ID is generated, the bisect is automatically triggered to locate **Logging in to the Test Environment** -1. Send an email to the compass-ci-reply(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. +1. Send an email to the compass-ci-robot(a)qq.com to [apply for an account](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/a…. 2. [Complete the environment configuration](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/ma… as instructed by the email. 3. Add the **sshd** field to the test task, submit the task, and [log in to the test environment](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu…. diff --git a/README.zh.md b/README.zh.md index 1339ef0..0e6cfb6 100644 --- a/README.zh.md +++ b/README.zh.md @@ -47,7 +47,7 @@ Compass-CI �� git repos�� **��** -1. �� compass-ci-reply(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� +1. �� compass-ci-robot(a)qq.com ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-account.md)�� 2. ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E6%9C%AC%E5%9C%B0%E5%AE%89%E8%A3%85compass-ci%E5%AE%A2%E6%88%B7%E7%AB%AF.md)�� 3. �� sshd ��[��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/%E5%A6%82%E4%BD%95%E7%99%BB%E5%BD%95%E6%B5%8B%E8%AF%95%E6%9C%BA%E8%B0%83%E6%B5%8B%E4%BB%BB%E5%8A%A1.md)�� -- 2.23.0

1 0

[PATCH compass-ci] lib/es_jobs.rb: fix error if jobs do not contain stats
by Wu Zhende 27 Jan '21

27 Jan '21

[Before] Traceback (most recent call last): 3: from /home/wuzhende/compass-ci/sbin/es-jobs:55:in `<main>' 2: from /home/wuzhende/compass-ci/lib/es_jobs.rb:167:in `output' 1: from /home/wuzhende/compass-ci/lib/es_jobs.rb:137:in `query_jobs_state' /home/wuzhende/compass-ci/lib/es_jobs.rb:99:in `initialize_result_hash': undefined method `each' for nil:NilClass (NoMethodError) [After] kvcount.category=benchmark: 1 kvcount.os=openeuler: 1 kvcount.suite=iperf: 1 kvcount.tbox_group=vm-2p16g: 1 raw.id.[arch=aarch64]: ["crystal.786605"] raw.id.[category=benchmark]: ["crystal.786605"] raw.id.[os=openeuler]: ["crystal.786605"] raw.id.[suite=iperf]: ["crystal.786605"] raw.id.[tbox_group=vm-2p16g]: ["crystal.786605"] Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/es_jobs.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index eb3f70d..e367a91 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -84,7 +84,7 @@ class ESJobs metrics.concat(stats.keys) end - metrics.uniq! + metrics.uniq end def initialize_result_hash(metrics) @@ -137,11 +137,11 @@ class ESJobs metrics = get_all_metrics(jobs) result = initialize_result_hash(metrics) jobs.each do |job| + kvcount(result, job) stats = job['stats'] next unless stats set_default_value(result, stats, metrics) - kvcount(result, job) end stats_count(result) -- 2.23.0

2 1

[PATCH v5 compass-ci] src/lib/job: add check runtime upper limit
by Luan Shengde 27 Jan '21

27 Jan '21

Add limit for borrowing machine: borrowing period <= 10 days Only jobs that runtime/sleep value is less than or equal to 10 days are allowed to borrow machines. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- src/lib/job.cr | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..5448082 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -143,6 +143,7 @@ class Job check_required_keys() check_account_info() + check_run_time() set_defaults() end @@ -401,6 +402,19 @@ class Job @hash.delete("my_token") end + private def check_run_time + return unless @hash.has_key?("ssh_pub_key") + + # runtime value is converted to second. + max_run_time = 10 * 24 * 3600 + + error_msg = "The maximum runtime should no more than 10 days." + + # case runtime value is more than 10 days, job will fail. + raise error_msg if hash["runtime"].to_i > max_run_time + raise error_msg if hash["sleep"].to_i > max_run_time + end + private def is_valid_account?(account_info) return false unless account_info.is_a?(JSON::Any) -- 2.23.0

2 1

[PATCH lkp-tests] distro/depends: fix install sysbench pkg depends
by Wei Jihui 27 Jan '21

27 Jan '21

[error] Error: Unable to find a match: mysql-devel [why] openeuler 20.03 has not libmysqlclient-dev: mysql-devel change it to libmariadbclient-dev Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/sysbench-dev | 2 +- distro/depends/sysbench-mysql-dev | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/distro/depends/sysbench-dev b/distro/depends/sysbench-dev index 3fa67deb..f570c463 100644 --- a/distro/depends/sysbench-dev +++ b/distro/depends/sysbench-dev @@ -1,2 +1,2 @@ pkg-config -libmysqlclient-dev +libmariadbclient-dev diff --git a/distro/depends/sysbench-mysql-dev b/distro/depends/sysbench-mysql-dev index ca5b03d8..65633b33 100644 --- a/distro/depends/sysbench-mysql-dev +++ b/distro/depends/sysbench-mysql-dev @@ -1,3 +1,2 @@ pkg-config -libmysqlclient-dev mysql-server -- 2.23.0

2 1

[PATCH compass-ci] providers/lib: introduce context.rb
by Xiao Shenwei 26 Jan '21

26 Jan '21

[why] expand erb templates need a class to save context [how] Context class provider a readable property @info, it will save the context required by erb, and a expand_erb method, it can expand a erb templates based on @info. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/lib/context.rb | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) create mode 100644 providers/lib/context.rb diff --git a/providers/lib/context.rb b/providers/lib/context.rb new file mode 100644 index 0000000..1ea36b0 --- /dev/null +++ b/providers/lib/context.rb @@ -0,0 +1,29 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'erb' +require_relative "#{ENV['LKP_SRC']}/lib/hashugar" + +# saves the context required by expand erb templates +class Context + attr_reader :info + + def initialize(mac, hostname, queues) + @info = { + 'mac' => mac, + 'hostname' => hostname, + 'queues' => queues + } + end + + def merge!(hash) + @info.merge!(hash) + end + + def expand_erb(template, context_hash = {}) + @info.merge!(context_hash) + context = Hashugar.new((a)info).instance_eval { binding } + ERB.new(template, nil, '%').result(context) + end +end -- 2.23.0

2 4

[PATCH lkp-tests] jobs: add multi-qemu.yaml multi-docker.yaml
by Xiao Shenwei 26 Jan '21

26 Jan '21

submit the job to a physical machine, then the machine will start some clients to consume job. Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- jobs/multi-docker.yaml | 17 +++++++++++++++++ jobs/multi-qemu.yaml | 23 +++++++++++++++++++++++ 2 files changed, 40 insertions(+) create mode 100644 jobs/multi-docker.yaml create mode 100644 jobs/multi-qemu.yaml diff --git a/jobs/multi-docker.yaml b/jobs/multi-docker.yaml new file mode 100644 index 000000000..8e42208a9 --- /dev/null +++ b/jobs/multi-docker.yaml @@ -0,0 +1,17 @@ +suite: multi-docker +category: functional +simplify-ci: +runtime: 100d +multi-docker-0: + nr_container: 20 + tbox_group: dc-1g +multi-docker-1: + nr_container: 10 + tbox_group: dc-2g +multi-docker-2: + nr_container: 5 + tbox_group: dc-4g +multi-docker-3: + nr_container: 20 + tbox_group: dc-8g +sleep: diff --git a/jobs/multi-qemu.yaml b/jobs/multi-qemu.yaml new file mode 100644 index 000000000..200f66065 --- /dev/null +++ b/jobs/multi-qemu.yaml @@ -0,0 +1,23 @@ +suite: multi-qemu +category: functional +simplify-ci: +runtime: 100d +multi-qemu-0: + nr_vm: 5 + tbox_group: vm-2p1g +multi-qemu-1: + nr_vm: 5 + tbox_group: vm-2p2g +multi-qemu-2: + nr_vm: 5 + tbox_group: vm-2p4g +multi-qemu-3: + nr_vm: 10 + tbox_group: vm-2p8g +multi-qemu-4: + nr_vm: 5 + tbox_group: vm-2p16g +multi-qemu-5: + nr_vm: 1 + tbox_group: vm-2p32g +sleep: -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/libvirt: add a libvirt provider
by Xiao Shenwei 26 Jan '21

26 Jan '21

request api: http://$SCHED_HOST:$SCHED_PORT/boot.libvirt/mac/$mac Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- providers/libvirt/run.rb | 60 ++++++++++++++++++++++++++++++++++++++++ 1 file changed, 60 insertions(+) create mode 100755 providers/libvirt/run.rb diff --git a/providers/libvirt/run.rb b/providers/libvirt/run.rb new file mode 100755 index 0000000..b4f8b4c --- /dev/null +++ b/providers/libvirt/run.rb @@ -0,0 +1,60 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'digest/md5' +require_relative "#{ENV['CCI_SRC']}/lib/log" +require_relative "#{ENV['CCI_SRC']}/lib/sched_client" +require_relative "#{ENV['CCI_SRC']}/providers/lib/context" + +def create_logger(hostname) + filename = "#{hostname}.log" + File.delete(filename) if FileTest.exist?(filename) + Log.new filename +end + +def compute_mac(string) + /(..)(..)(..)(..)(..)/ =~ Digest::MD5.hexdigest(string) + "0a:#{$1}:#{$2}:#{$3}:#{$4}:#{$5}" +end + +def job_exist?(response) + flag = false + if response['job_id'].empty? + puts '----------' + puts 'No job now' + puts '----------' + flag = true + end + return flag +end + +def request_job(context, sched_client, logger) + mac = context.info['mac'] + hostname = context.info['hostname'] + queues = context.info['queues'] + sched_client.register_mac2host(hostname, mac) + sched_client.register_host2queues(hostname, queues) + response = JSON.parse(sched_client.consume_job('libvirt', 'mac', mac)) + if job_exist?(response) + logger.info('No job now') + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) + response = nil + end + return response +end + +def main(hostname, queues) + logger = create_logger(hostname) + mac = compute_mac hostname + context = Context.new(mac, hostname, queues) + sched_client = SchedClient.new + response = request_job(context, sched_client, logger) + return if response.nil? + + sched_client.delete_mac2host(mac) + sched_client.delete_host2queues(hostname) +end -- 2.23.0

1 0