- Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 08 Mar '21

08 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..53c9e59 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -11,6 +11,7 @@ require 'priority_queue' require 'English' require 'elasticsearch' require_relative 'constants.rb' +require_relative 'json_logger.rb' # worker threads class GitMirror @@ -106,6 +107,7 @@ class MirrorMain @git_info = {} @defaults = {} @git_queue = Queue.new + @log = JSONLogger.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") clone_upstream_repo load_fork_info @@ -192,6 +194,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) end def do_push(fork_key) @@ -509,4 +512,13 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, nr_new_branch) + @log.info({ + msg: 'new refs', + time: Time.now, + repo: git_repo, + nr_new_branch: nr_new_branch + }.to_json) + end end -- 2.23.0

1 0

[PATCH v2 lkp-tests] /stats add proc-vmstat.rb
by Lu Weitao 08 Mar '21

08 Mar '21

[Why] convert stats/proc-vmstat to proc_vmstat() of proc-vmstat.rb Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- stats/proc-vmstat.rb | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 stats/proc-vmstat.rb diff --git a/stats/proc-vmstat.rb b/stats/proc-vmstat.rb new file mode 100644 index 000000000..cd252faf1 --- /dev/null +++ b/stats/proc-vmstat.rb @@ -0,0 +1,10 @@ + +def proc_vmstat(log_lines) + result = Hash.new { |hash, key| hash[key] = [] } + log_lines.each do |line| + key, value = line.split(/:?\s+/) + result[key] << value.to_i + end + + result +end -- 2.23.0

2 2

[PATCH v2 compass-ci 2/2] lib/json_logger.rb: fix that no warp cause no puts
by Li Yuanchao 05 Mar '21

05 Mar '21

The log didn't warp before, that caused sub-fluentd can not put the log out. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/json_logger.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/json_logger.rb b/lib/json_logger.rb index 78f95ca..2c33c2f 100644 --- a/lib/json_logger.rb +++ b/lib/json_logger.rb @@ -29,7 +29,7 @@ class JSONLogger < Logger msg = { 'message' => msg } unless msg.is_a?(Hash) logger_hash.merge!(msg) - logger_hash.to_json + logger_hash.to_json + "\n" } def initialize(logdev = STDOUT, formatter = FORMATTER) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: Add new_refs_log for kibana
by Li Yuanchao 05 Mar '21

05 Mar '21

The logs can be shown as chart by kibana. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bb0c9f9..ae7e64b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -11,6 +11,7 @@ require 'priority_queue' require 'English' require 'elasticsearch' require_relative 'constants.rb' +require_relative 'json_logger.rb' # worker threads class GitMirror @@ -192,6 +193,7 @@ class MirrorMain feedback_info[:new_refs] = new_refs send_message(feedback_info) + new_refs_log(git_repo, new_refs[:heads].length) end def do_push(fork_key) @@ -509,4 +511,14 @@ class MirrorMain traverse_repodir(repodir, belong) end end + + def new_refs_log(git_repo, nr_new_branch) + log = JSONLogger.new + log.info({ + msg: 'new refs', + time: Time.now, + repo: git_repo, + nr_new_branch: nr_new_branch + }.to_json) + end end -- 2.23.0

1 0

[PATCH v5 lkp-tests] daemon: add mysql
by Wei Jihui 05 Mar '21

05 Mar '21

add mysql for sysbench-mysql test Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 91 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 91 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..808e3eec --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,91 @@ +#!/bin/sh +# - data_dev +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_test}" +: "${mysql_password:=mysql_password}" + +prepare() +{ + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 + systemctl start tuned >/dev/null 2>&1 +} + +insmod_getcpu() +{ + if [ -d "/getcpu" ]; then + cd /getcpu && insmod getcpu-hi.ko + else + echo "this test will not use getcpu optimize to bind core." + fi +} + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data + + mkdir -p /data/mysql/{data,share,tmp,run,log} + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +configure_mysql() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +start_mysql() +{ + numactl -C 0-92 -i 0-3 "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + prepare + insmod_getcpu + create_basedir + configure_mysql + create_database + start_mysql +} +main -- 2.23.0

1 0

[PATCH v5 compass-ci] doc/manual: add document for submit container
by Luan Shengde 05 Mar '21

05 Mar '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.en.md | 63 ++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 doc/manual/build-lkp-tests-container.en.md diff --git a/doc/manual/build-lkp-tests-container.en.md b/doc/manual/build-lkp-tests-container.en.md new file mode 100644 index 0000000..47c85c5 --- /dev/null +++ b/doc/manual/build-lkp-tests-container.en.md @@ -0,0 +1,63 @@ +# Preface + +We provide a docker container to suit various of Linux OS(es). +In this case you do not need to install the lkp-tests to your local server. +Also you can avoid installation failures from undesired dependency package(s). + +# Prepare + + Do on your local server: + + - install docker + - apply account and config default yaml + - generate ssh key(s) + +# build container + +## 1. download resource + + Download lkp-tests and compass-ci to your local server. + + Command(s): + + git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git + +## 2. add environment variable(s) + + Command(s): + + echo "export LKP_SRC=$PWD/lkp-tests" >> ~/.${SHELL##*/}rc + echo "export CCI_SRC=$PWD/compass-ci" >> ~/.${SHELL##*/}rc + source ~/.${SHELL##*/}rc + +## 3. build docker image + + Command(s): + + cd compass-ci/container/submit + ./build + +## 4. add executable file + + Command(s): + + ln -s $CCI_SRC/container/submit/submit /usr/bin/submit + +# try it + + instruction: + + You can directly use the command 'submit' to submit jobs. + It is the same as you install the lkp-tests on your own server. + It will start a disposable container to submit your job. + The container will attach the directory lkp-tests to the container itself. + You can edit the job yaml(s) in lkp-tests/jobs and it will take effect when you submit jobs. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + About submit: + + For detailed usage for command submit, please reference to: [submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… -- 2.23.0

1 0

[PATCH lkp-tests 08/11] install-iso-pre: add step: prepare_submit
by Yu Chuan 05 Mar '21

05 Mar '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- distro/depends/install-iso-pre | 2 ++ tests/install-iso-pre | 18 +++++++++++++++++- 2 files changed, 19 insertions(+), 1 deletion(-) diff --git a/distro/depends/install-iso-pre b/distro/depends/install-iso-pre index 6f88eb919896..4569e1e3cdc0 100644 --- a/distro/depends/install-iso-pre +++ b/distro/depends/install-iso-pre @@ -1 +1,3 @@ lvm2 +ruby +git diff --git a/tests/install-iso-pre b/tests/install-iso-pre index e27c9ea80232..d867f1441003 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -163,7 +163,23 @@ prepare_ks() prepare_submit() { - : + ${LKP_SRC}/sbin/install-dependencies.sh + + export CCI_REPOS="/c" + + mkdir -p $CCI_REPOS && cd $CCI_REPOS + git clone $LAB_Z9_URL || die "clone lab-z9 git repo failed: $LAB_Z9_URL" + + local config_yaml="/etc/compass-ci/defaults/sparrow.yaml" + mkdir -p $(dirname $config_yaml) + cat >> $config_yaml <<-EOF + SCHED_HOST: ${SCHED_HOST} + SCHED_PORT: ${SCHED_PORT} + my_name: ${my_name} + my_email: ${my_email} + my_token: ${MY_TOKEN} + lab: z9 + EOF } submit_install_iso_yaml() -- 2.23.0

1 0

[PATCH lkp-tests 01/11] feat: step 1 of iso2rootfs: install iso to testbox and reboot
by Yu Chuan 05 Mar '21

05 Mar '21

[Why] When testbox request job in pxe -> ipxe step, scheduler returns a response composed of ipxe cmd. So we add 'custom_ipxe' field of job.yaml, then we can customize the response what testbox get, then testbox can auto install iso by executing the specified ipxe cmd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso.yaml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 jobs/install-iso.yaml diff --git a/jobs/install-iso.yaml b/jobs/install-iso.yaml new file mode 100644 index 000000000000..bcccc6b76f85 --- /dev/null +++ b/jobs/install-iso.yaml @@ -0,0 +1,3 @@ +suite: install-iso + +custom_ipxe: -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/job.cr: delete os_mount in common_dir for upload rpm
by Li Ping 05 Mar '21

05 Mar '21

[why] rpm has no much to do with os_mount. Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..c07033c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -267,14 +267,19 @@ class Job pkg_style = JSON.parse("{}") if pkg_style == nil - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + tmp_os = pkg_style["os"]? || "#{os}" + tmp_os_arch = pkg_style["os_arch"]? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"]? || "#{os_version}" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + mount_type = pkg_style["os_mount"]? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

2 2

[PATCH v2 lkp-tests 1/2] jobs/iso2rootfs.yaml: change some field to secrets field.
by Li Ping 05 Mar '21

05 Mar '21

[why] the related fields when submit test.yaml should be add to secrets field. submit_name: submit_email: submit_token: and the required fields when submit test.yaml not have much to do with the iso2rootfs be changed to level-1 parameter. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/iso2rootfs.yaml | 33 +++++++++++++++++---------------- 1 file changed, 17 insertions(+), 16 deletions(-) diff --git a/jobs/iso2rootfs.yaml b/jobs/iso2rootfs.yaml index 55c4345fa..5df2cdf61 100644 --- a/jobs/iso2rootfs.yaml +++ b/jobs/iso2rootfs.yaml @@ -51,23 +51,24 @@ iso2rootfs: # http://1.1.1.1//dailybuilds/1970-01-01-00-00-00/openEuler-20.03-LTS-aarch64… dailybuild_iso_url_file: - ################# - # submit test yaml related fields - ################# +################# +# submit test yaml related fields +################# - # three required fields when submit test yaml - submit_name: +# three required fields when submit test yaml +secrets: + submit_name: submit_email: submit_token: - # submit target tests yaml - ## 1. You can add as many jobs as you like. - ## 2. The following three fields is required for every test job. - test1_yaml: - test1_os_mount: - test1_testbox: - ## the following three fields is required when your job.yaml and - ## script for this test are from the internet. - test1_git_url: - test1_git_yaml: - test1_git_script: +# submit target tests yaml +## 1. You can add as many jobs as you like. +## 2. The following three fields is required for every test job. +test1_yaml: +test1_os_mount: +test1_testbox: +## the following three fields is required when your job.yaml and +## script for this test are from the internet. +test1_git_url: +test1_git_yaml: +test1_git_script: -- 2.23.0

1 1