- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib: parse mail list from yaml by type as to and bcc mail list
by Cao Xueliang 03 Mar '21

03 Mar '21

Yaml file define: delimiter: to: mai1, mail3 bcc: mail2, mail3 The mail_bisect_result calls parse_mail_list('delimiter') to get the delimiter mail list. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 14 +++++++++++--- lib/mail_list.yaml | 3 +++ lib/parse_mail_list.rb | 13 +++++++++++++ 3 files changed, 27 insertions(+), 3 deletions(-) create mode 100644 lib/mail_list.yaml create mode 100644 lib/parse_mail_list.rb diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 104f098..23a8ade 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -6,6 +6,7 @@ require_relative 'git' require_relative 'es_query' require_relative 'constants' require_relative 'mail_client' +require_relative 'parse_mail_list' require_relative 'assign_account_client' # compose and send email for bisect result @@ -22,10 +23,17 @@ class MailBisectResult @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email # now send mail to review - @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + @bcc = 'caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + end + + def parse_mail_info + mail_hash = parse_mail_list('delimiter') + @to = mail_hash['to'] if mail_hash.key?('to') + @bcc = mail_hash['bcc'] if mail_hash.key?('bcc') end def create_send_email + parse_mail_info send_report_mail(compose_mail) send_account_mail rm_work_dir @@ -35,7 +43,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' - bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" + report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" @@ -58,7 +66,7 @@ class MailBisectResult #{pkgbuild_repo_url} #{first_bad_commit_job_url} #{bisect_job_url} - #{bisect_report_doc} + #{report_doc} Regards, Compass CI team BODY diff --git a/lib/mail_list.yaml b/lib/mail_list.yaml new file mode 100644 index 0000000..d982c93 --- /dev/null +++ b/lib/mail_list.yaml @@ -0,0 +1,3 @@ +delimiter: + to: caoxl78320(a)163.com + bcc: caoxl78320(a)163.com, wfg(a)mail.ustc.edu.cn, ljx.joe(a)qq.com, huming15(a)163.com diff --git a/lib/parse_mail_list.rb b/lib/parse_mail_list.rb new file mode 100644 index 0000000..49b5c88 --- /dev/null +++ b/lib/parse_mail_list.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'yaml' + +def parse_mail_list(type) + content = {} + mail_list_yaml = './mail_list.yaml' + content = YAML.safe_load(File.open(mail_list_yaml)) if FileTest.exists?(mail_list_yaml) + return {} unless content[type] + + return content[type] +end -- 2.23.0

2 5

[PATCH compass-ci] src/stats_worker:delete filter error_id and add log
by Cao Xueliang 02 Mar '21

02 Mar '21

We add the filter error_id in commit: 1c154d888fab1fce4623663b4a88b9a1630171c2, we don't need it now, so delete the filter error_id, add the new_error_id log for statistical. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/stats_worker.cr | 17 ++++++----------- 1 file changed, 6 insertions(+), 11 deletions(-) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index 01bae21..836d1f8 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -82,25 +82,20 @@ class StatsWorker } ) - error_id = select_error_id(check_new_error_ids(error_ids, job_id)) - if error_id + new_error_ids = check_new_error_ids(error_ids, job_id) + unless new_error_ids.empty? + sample_error_id = new_error_ids.sample STDOUT.puts "send a delimiter task: job_id is #{job_id}" - @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => error_id, + @tq.add_task(DELIMITER_TASK_QUEUE, JSON.parse({"error_id" => sample_error_id, "job_id" => job_id, "lab" => LAB}.to_json)) + msg = %({"job_id": "#{job_id}", "new_error_id": "#{sample_error_id}"}) + system "echo '#{msg}'" end msg = %({"job_id": "#{job_id}", "job_state": "extract_finished"}) system "echo '#{msg}'" end - def select_error_id(new_error_ids : Array) - new_error_ids.each do |error_id| - return error_id if /(cpp|c|h):(warning|error)/i =~ error_id - end - - return nil - end - def check_new_error_ids(error_ids : Array, job_id : String) new_error_ids = [] of String error_ids.each do |error_id| -- 2.23.0

1 0

[PATCH v3 compass-ci 4/4] container/dracut-initrd: add 90lkp module into Dockerfile
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/Dockerfile | 2 ++ 1 file changed, 2 insertions(+) diff --git a/container/dracut-initrd/Dockerfile b/container/dracut-initrd/Dockerfile index 7f8448f..b8c551f 100644 --- a/container/dracut-initrd/Dockerfile +++ b/container/dracut-initrd/Dockerfile @@ -6,6 +6,8 @@ FROM debian MAINTAINER Wu Fenguang <wfg(a)mail.ustc.edu.cn> ADD bin /usr/local/bin +ADD modules.d /usr/lib/modules.d +COPY modules.d/90lkp /usr/lib/dracut/modules.d/90lkp COPY conf/sources.list* /etc/apt/ COPY conf/add-modules.conf /etc/dracut.conf.d/ WORKDIR /usr/local/bin -- 2.23.0

1 0

[PATCH v3 compass-ci 2/4] container/dracut-initrd: README of module 90lkp
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/modules.d/90lkp/README | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 container/dracut-initrd/modules.d/90lkp/README diff --git a/container/dracut-initrd/modules.d/90lkp/README b/container/dracut-initrd/modules.d/90lkp/README new file mode 100644 index 0000000..7d7a205 --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/README @@ -0,0 +1,5 @@ +dracut lkp module + +This module is to deploy lkp module into rootfs. +Module lkp(Linux Kernel Performance) is a test framework, which is used +to make self-defined test for os kernel. -- 2.23.0

1 0

[PATCH v3 compass-ci 1/4] container/dracut-initrd: add module 90lkp/module-setup.sh
by Xu Xijian 02 Mar '21

02 Mar '21

90lkp module a module to deploy lkp module into rootfs. module-setup.sh is a base setup script for every kernel module. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../modules.d/90lkp/module-setup.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/dracut-initrd/modules.d/90lkp/module-setup.sh diff --git a/container/dracut-initrd/modules.d/90lkp/module-setup.sh b/container/dracut-initrd/modules.d/90lkp/module-setup.sh new file mode 100755 index 0000000..c188600 --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/module-setup.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +check() { + return 0 +} + +depends() { + # We do not depend on any modules - just some root + return 0 +} + +# called by dracut +installkernel() { + return 0 +} + +install() { + inst_hook pre-pivot 10 "$moddir/lkp-deploy.sh" +} -- 2.23.0

1 0

[PATCH compass-ci 2/2] test2
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/test2 | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 container/dracut-initrd/test2 diff --git a/container/dracut-initrd/test2 b/container/dracut-initrd/test2 new file mode 100644 index 0000000..e69de29 -- 2.23.0

1 1

[PATCH compass-ci 1/2] test1
by Xu Xijian 02 Mar '21

02 Mar '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/test1 | 0 1 file changed, 0 insertions(+), 0 deletions(-) create mode 100644 container/dracut-initrd/test1 diff --git a/container/dracut-initrd/test1 b/container/dracut-initrd/test1 new file mode 100644 index 0000000..e69de29 -- 2.23.0

1 1

[PATCH v2 compass-ci 2/2] container/dracut-initrd: add module 90lkp/lkp-deploy.sh
by Xu Xijian 02 Mar '21

02 Mar '21

lkp-deploy.sh does the concrete work of 90lkp, it adds the lkp scripts into testbox's rootfs to run test job. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../modules.d/90lkp/lkp-deploy.sh | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100755 container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh diff --git a/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh new file mode 100755 index 0000000..e88959b --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/lkp-deploy.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# SPDX-License-Identifier: GPL-2.0 + +. /lib/dracut-lib.sh + +# transfer LKP dirs +[ -d /lkp ] || return 0 + +cp -a /lkp "$NEWROOT"/ + +for i in /lkp/lkp/src/rootfs/addon/* /usr/local/* /opt/* +do + dir=$(basename "$i") + + [ "$i" != "${i%/\*}" ] && continue # skip: i='dir/*' + + [ -d "$NEWROOT/$dir" ] || mkdir -p "$NEWROOT/$dir" + + for j in "$i"/* "$i"/.??* + do + [ "$j" != "${j%/\*}" ] && continue # skip: j='dir/*' + + [ -f "$j" ] && { + cp -a "$j" "$NEWROOT/$dir"/ + continue + } + + subdir=$(basename "$j") + + [ -d "$NEWROOT/$dir/$subdir" ] || mkdir -p "$NEWROOT/$dir/$subdir" + + for k in "$j"/* + do + [ "$k" != "${k%/\*}" ] && continue # skip: k='dir/*' + + cp -a "$j"/* "$NEWROOT/$dir/$subdir"/ + done + done +done + +[ -d /usr/src ] && + cp -a /usr/src "$NEWROOT"/usr/ + +kmdir=/lib/modules/$(uname -r) +if test -d "$kmdir" && ! test -d "$NEWROOT/$kmdir"; then + cp -an "$kmdir" "$NEWROOT"/lib/modules/ + cp -an /lib/firmware "$NEWROOT"/lib/ +fi -- 2.23.0

2 2

[PATCH v3 lkp-tests] daemon: add mysql
by Wei Jihui 02 Mar '21

02 Mar '21

add mysql for sysbench-mysql test Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 92 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 92 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..a275a812 --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,92 @@ +#!/bin/sh +# - nic +# - device +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +prepare() +{ + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 + systemctl start tuned >/dev/null 2>&1 +} + +insmod_getcpu() +{ + if [ -d "/getcpu" ]; then + cd /getcpu && insmod getcpu-hi.ko + else + echo "this test will not use getcpu optimize to bind core." + fi +} + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$device" + mount /dev/"$device" /data + + mkdir -p /data/mysql/data /data/mysql/share /data/mysql/tmp /data/mysql/run /data/mysql/log + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + numactl -C 0-92 -i 0-3 "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + prepare + insmod_getcpu + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 0

[PATCH compass-ci 4/4] sparrow/5-build/register-account: check account info before register
by Liu Yinsi 02 Mar '21

02 Mar '21

for user locally deploy compass-ci client, if not fill config yaml $CCI_SRC/sparrow/setup.yaml, failed to register account, so check my_email, my_name. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/register-account | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/sparrow/5-build/register-account b/sparrow/5-build/register-account index 1a23e07..94b7407 100755 --- a/sparrow/5-build/register-account +++ b/sparrow/5-build/register-account @@ -2,4 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -$CCI_SRC/sbin/build-my-info.rb +source ${CCI_SRC}/lib/log.sh + +if [ -n "$my_email" ] && [ -n "$my_name" ]; then + $CCI_SRC/sbin/build-my-info.rb -e $my_email -n $my_name +else + log_info "if you need to register account, please try: $CCI_SRC/sbin/build-my-info.rb -e \$my_email -n \$my_name" +fi -- 2.23.0

1 0