March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] daemon: add mysql
by Wei Jihui 02 Mar '21

02 Mar '21

add mysql for sysbench-mysql test Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..a275a812 --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,92 @@ +#!/bin/sh +# - nic +# - device +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +prepare() +{ + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 + systemctl start tuned >/dev/null 2>&1 +} + +insmod_getcpu() +{ + if [ -d "/getcpu" ]; then + cd /getcpu && insmod getcpu-hi.ko + else + echo "this test will not use getcpu optimize to bind core." + fi +} + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$device" + mount /dev/"$device" /data + + mkdir -p /data/mysql/data /data/mysql/share /data/mysql/tmp /data/mysql/run /data/mysql/log + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + numactl -C 0-92 -i 0-3 "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + prepare + insmod_getcpu + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 0

[PATCH compass-ci 4/4] sparrow/5-build/register-account: check account info before register
by Liu Yinsi 02 Mar '21

02 Mar '21

for user locally deploy compass-ci client, if not fill config yaml $CCI_SRC/sparrow/setup.yaml, failed to register account, so check my_email, my_name. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/register-account | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/sparrow/5-build/register-account b/sparrow/5-build/register-account index 1a23e07..94b7407 100755 --- a/sparrow/5-build/register-account +++ b/sparrow/5-build/register-account @@ -2,4 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/build-my-info.rb +source ${CCI_SRC}/lib/log.sh + +if [ -n "$my_email" ] && [ -n "$my_name" ]; then + $CCI_SRC/sbin/build-my-info.rb -e $my_email -n $my_name +else + log_info "if you need to register account, please try: $CCI_SRC/sbin/build-my-info.rb -e \$my_email -n \$my_name" +fi -- 2.23.0

1 0

[PATCH compass-ci 3/4] sparrow/3-code/dev-env: delete redunbant code
by Liu Yinsi 02 Mar '21

02 Mar '21

0-package/read-config already copy $CCI_SRC/sparrow/setup.yaml to /etc/compass-ci/setup.yaml, so delete redunbant code. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 5 ----- 1 file changed, 5 deletions(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 698a4a2..df7b1ce 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -2,12 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -mkdir -p /etc/compass-ci -cp -a $CCI_SRC/sparrow/setup.yaml /etc/compass-ci/setup.yaml - -server_ip=$(awk '/^server_ip:\s/ {print $2; exit}' /etc/compass-ci/setup.yaml) [ -z $server_ip ] && server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') - server_name=$(hostname | cut -f1 -d.) : ${sched_host:=$server_ip} : ${sched_port:=3000} -- 2.23.0

1 0

[PATCH compass-ci 3/3] lib/job.cr: set job_state when submit job
by Wu Zhende 02 Mar '21

02 Mar '21

Set job_state to "submit" when submitting a job. Because each phase corresponds to a state in lifecycle. We can query all submitted jobs that have not been consumed in ES based on this status. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/lib/job.cr b/src/lib/job.cr index 63ec8fb..60e0b65 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -142,6 +142,7 @@ class Job # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" @hash["id"] = JSON::Any.new("#{id}") + self["job_state"] = "submit" check_required_keys() check_account_info() -- 2.23.0

1 0

[PATCH compass-ci 2/3] lib/mq.cr: add rabbitmq class
by Wu Zhende 02 Mar '21

02 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/mq.cr | 39 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 src/lib/mq.cr diff --git a/src/lib/mq.cr b/src/lib/mq.cr new file mode 100644 index 0000000..011d732 --- /dev/null +++ b/src/lib/mq.cr @@ -0,0 +1,39 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +require "singleton" +require "amqp-client" + +class MQClient + getter ch : AMQP::Client::Channel + MQ_HOST = ENV.has_key?("MQ_HOST") ? ENV["MQ_HOST"] : "172.17.0.1" + MQ_PORT = (ENV.has_key?("MQ_PORT") ? ENV["MQ_PORT"] : 5672).to_i32 + + def initialize(host = MQ_HOST, port = MQ_PORT) + @client = AMQP::Client.new("amqp://#{host}:#{port}") + conn = @client.connect + @ch = conn.channel.as(AMQP::Client::Channel) + end + + def self.instance + Singleton::Of(self).instance + end + + def pushlish_confirm(queue, msg) + q = @ch.queue(queue) + q.publish_confirm msg + end + + def pushlish(queue, msg) + q = @ch.queue(queue) + q.publish msg + end + + def get(queue) + q = @ch.queue(queue) + q.subscribe(no_ack: false) do |msg| + @ch.basic_ack(msg.delivery_tag) + msg.body_io.to_s + end + end +end -- 2.23.0

1 0

[PATCH compass-ci 1/3] container/scheduler: add start depend
by Wu Zhende 02 Mar '21

02 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/scheduler/start-depends | 1 + 1 file changed, 1 insertion(+) diff --git a/container/scheduler/start-depends b/container/scheduler/start-depends index 4a17b46..1aced31 100755 --- a/container/scheduler/start-depends +++ b/container/scheduler/start-depends @@ -1,3 +1,4 @@ redis es taskqueue +rabbitmq -- 2.23.0

1 0

[PATCH v2 lkp-tests] lib/bootstrap.sh: change $HOSTNAME suffix
by Wang Yong 02 Mar '21

02 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/bootstrap.sh | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 76f07e6a7..52c9da32a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -273,6 +273,9 @@ setup_hosts() { is_docker && return + suffix="sh.intel.com" + [ -n "${CCI_SRC}" ] && suffix="compass-ci.net" + # /etc/hosts may be shared when it's NFSROOT and there is no obvious # way to detect if rootfs is already RAM based. So unconditionally # symlink it to my own tmpfs copy. @@ -283,8 +286,8 @@ setup_hosts() else cp /etc/hosts $tmpfs_hosts fi - echo "127.0.0.1 $HOSTNAME.sh.intel.com $HOSTNAME" >> $tmpfs_hosts - echo "::1 $HOSTNAME.sh.intel.com $HOSTNAME" >> $tmpfs_hosts + echo "127.0.0.1 $HOSTNAME.${suffix} $HOSTNAME" >> $tmpfs_hosts + echo "::1 $HOSTNAME.${suffix} $HOSTNAME" >> $tmpfs_hosts ln -fs $tmpfs_hosts /etc/hosts } -- 2.23.0

1 0

[PATCH compass-ci] scheduler/find_job_boot.cr: fix tbox state if job interruption
by Wei Jihui 02 Mar '21

02 Mar '21

tbox state is running when job interruption, this tbox will always rebooting if no new job, so update tbox state when it requests job every time. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/lib/sched.cr | 12 ++++++++++++ src/scheduler/find_job_boot.cr | 1 + 2 files changed, 13 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index dfec365..1b2836f 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -155,6 +155,18 @@ class Sched @es.update_tbox(job["testbox"], hash) end + def initialize_tbox_boot_wtmp(testbox) + testbox = testbox.to_s + time = Time.local.to_s("%Y-%m-%d %H:%M:%S") + + hash = { + "time" => time, + "state" => "rebooting", + } + + @redis.update_wtmp(testbox, hash) + end + def report_ssh_port testbox = @env.params.query["tbox_name"] ssh_port = @env.params.query["ssh_port"].to_s diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 8dd22a8..2024843 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -16,6 +16,7 @@ class Sched host = value end + initialize_tbox_boot_wtmp(host) response = get_job_boot(host, boot_type) job_id = response[/tmpfs\/(.*)\/job\.cgz/, 1]? @log.info(%({"job_id": "#{job_id}", "job_state": "boot"})) if job_id -- 2.23.0

1 0

[PATCH v4 compass-ci] lib: parse mail list from yaml by type as to and bcc mail list
by Cao Xueliang 02 Mar '21

02 Mar '21

Yaml file define: delimiter: to: mail1, mail3 bcc: mail2, mail3 The mail_bisect_result calls parse_mail_list('delimiter') to get the delimiter mail list. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 15 +++++++++++---- lib/parse_mail_list.rb | 12 ++++++++++++ 2 files changed, 23 insertions(+), 4 deletions(-) create mode 100644 lib/parse_mail_list.rb diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 104f098..503ed43 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -6,6 +6,7 @@ require_relative 'git' require_relative 'es_query' require_relative 'constants' require_relative 'mail_client' +require_relative 'parse_mail_list' require_relative 'assign_account_client' # compose and send email for bisect result @@ -21,11 +22,17 @@ class MailBisectResult @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email - # now send mail to review - @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + end + + def parse_mail_info + mail_hash = parse_mail_list('delimiter') + @to = mail_hash['to'] if mail_hash.key?('to') + @bcc = mail_hash['bcc'] if mail_hash.key?('bcc') + raise 'Need to add bcc email for bisect report.' unless @bcc end def create_send_email + parse_mail_info send_report_mail(compose_mail) send_account_mail rm_work_dir @@ -35,7 +42,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' - bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" + report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" @@ -58,7 +65,7 @@ class MailBisectResult #{pkgbuild_repo_url} #{first_bad_commit_job_url} #{bisect_job_url} - #{bisect_report_doc} + #{report_doc} Regards, Compass CI team BODY diff --git a/lib/parse_mail_list.rb b/lib/parse_mail_list.rb new file mode 100644 index 0000000..c98a466 --- /dev/null +++ b/lib/parse_mail_list.rb @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'yaml' + +def parse_mail_list(type) + content = {} + mail_list_yaml = '/etc/compass-ci/report-email.yaml' + content = YAML.safe_load(File.open(mail_list_yaml)) if FileTest.exists?(mail_list_yaml) + + return content[type] || {} +end -- 2.23.0

1 0

[PATCH lkp-tests] PACKAGER use default value "Compass-CI" when use makepkg
by Lin Jiaxin 02 Mar '21

02 Mar '21

If the value of PACKAGER is not specified in job.yaml, set it to the default value of "Compass-CI". [Now] % cat .PKGINFO | grep packager packager = Unknown Packager [After] % cat .PKGINFO | grep packager packager = Compass-CI Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- tests/build-pkg | 2 +- tests/cci-makepkg | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index f3545a19..6a083a9b 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -159,7 +159,7 @@ build_source_pkg() export V=1 - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" PACKAGER="${PACKAGER:-Compass-CI}" \ $LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf local build_state="$?" diff --git a/tests/cci-makepkg b/tests/cci-makepkg index ecbf7213..556de6b0 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -142,7 +142,7 @@ build_source_pkg() local makepkg="$LKP_SRC/sbin/makepkg ${pkg_args} --config $LKP_SRC/etc/makepkg.conf" echo "${makepkg}" - PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" $makepkg + PACMAN="${mark}" BUILDDIR=$TMP CARCH=$arch PKGEXT=.cgz CGZDEST="$sync_dest/$cgz_name" PACKAGER="${PACKAGER:-Compass-CI}" $makepkg [ "$?" == 0 ] || exit 1 update_softlink -- 2.23.0

1 0