- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] lib/es_jobs.rb: don't sum '.message'
by Wang Chenglong 02 Mar '21

02 Mar '21

[Why] The 'xxx.message' in es-jobs just show the error message, it doesn't has to be summed. before: sum.stats.ansible_test.error.No-package-yum-utils-available.message: 1 sum.stats.ansible_test.error.No-package-yum-utils-available: 1 raw.stats.ansible_test.error.No-package-yum-utils-available.message: ["{\"ansible_loop_var\": \"item\", \"changed\": false, \"failures\": [\"No package yum-utils available.\"], \"item\": [\"yum-utils\", \"device-mapper-persistent-data\", \"lvm2\"], \"msg\": \"Failed to install some of the specified packages\", \"rc\": 1, \"results\": []}"] raw.stats.ansible_test.error.No-package-yum-utils-available: [1] after: sum.stats.ansible_test.error.No-package-yum-utils-available: 1 raw.stats.ansible_test.error.No-package-yum-utils-available.message: ["{\"ansible_loop_var\": \"item\", \"changed\": false, \"failures\": [\"No package yum-utils available.\"], \"item\": [\"yum-utils\", \"device-mapper-persistent-data\", \"lvm2\"], \"msg\": \"Failed to install some of the specified packages\", \"rc\": 1, \"results\": []}"] raw.stats.ansible_test.error.No-package-yum-utils-available: [1] Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/es_jobs.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/lib/es_jobs.rb b/lib/es_jobs.rb index c5c0f0d..b0d0fb2 100644 --- a/lib/es_jobs.rb +++ b/lib/es_jobs.rb @@ -124,6 +124,8 @@ class ESJobs def stats_count(result) result['raw.stats'].each do |key, value| + next if key.end_with?('.message') + if function_stat?(key) result['sum.stats'][key] = value.compact.size else -- 2.23.0

2 2

[PATCH lkp-tests] lib/bootstrap.sh: change $HOSTNAME suffix
by Wang Yong 02 Mar '21

02 Mar '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/bootstrap.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 76f07e6a7..b2ded1893 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -283,8 +283,8 @@ setup_hosts() else cp /etc/hosts $tmpfs_hosts fi - echo "127.0.0.1 $HOSTNAME.sh.intel.com $HOSTNAME" >> $tmpfs_hosts - echo "::1 $HOSTNAME.sh.intel.com $HOSTNAME" >> $tmpfs_hosts + echo "127.0.0.1 $HOSTNAME.compass-ci.net $HOSTNAME" >> $tmpfs_hosts + echo "::1 $HOSTNAME.compass-ci.net $HOSTNAME" >> $tmpfs_hosts ln -fs $tmpfs_hosts /etc/hosts } -- 2.23.0

2 2

[PATCH compass-ci] container/srv-http: add /srv/initrd/rpmbuild-pkg
by Li Ping 02 Mar '21

02 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- container/srv-http/start-initrd | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 7c73ea1..495cc25 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -12,6 +12,7 @@ cmd=( -p 11303:11300 -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro + -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro -v /etc/localtime:/etc/localtime:ro -d srv-http -- 2.23.0

2 1

[PATCH v2 compass-ci] lib: parse mail list from yaml by type as to and bcc mail list
by Cao Xueliang 02 Mar '21

02 Mar '21

Yaml file define: delimiter: to: mai1, mail3 bcc: mail2, mail3 The mail_bisect_result calls parse_mail_list('delimiter') to get the delimiter mail list. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 14 +++++++++++--- lib/mail_list.yaml | 3 +++ lib/parse_mail_list.rb | 12 ++++++++++++ 3 files changed, 26 insertions(+), 3 deletions(-) create mode 100644 lib/mail_list.yaml create mode 100644 lib/parse_mail_list.rb diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 104f098..23a8ade 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -6,6 +6,7 @@ require_relative 'git' require_relative 'es_query' require_relative 'constants' require_relative 'mail_client' +require_relative 'parse_mail_list' require_relative 'assign_account_client' # compose and send email for bisect result @@ -22,10 +23,17 @@ class MailBisectResult @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email # now send mail to review - @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + @bcc = 'caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + end + + def parse_mail_info + mail_hash = parse_mail_list('delimiter') + @to = mail_hash['to'] if mail_hash.key?('to') + @bcc = mail_hash['bcc'] if mail_hash.key?('bcc') end def create_send_email + parse_mail_info send_report_mail(compose_mail) send_account_mail rm_work_dir @@ -35,7 +43,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' - bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" + report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" @@ -58,7 +66,7 @@ class MailBisectResult #{pkgbuild_repo_url} #{first_bad_commit_job_url} #{bisect_job_url} - #{bisect_report_doc} + #{report_doc} Regards, Compass CI team BODY diff --git a/lib/mail_list.yaml b/lib/mail_list.yaml new file mode 100644 index 0000000..d982c93 --- /dev/null +++ b/lib/mail_list.yaml @@ -0,0 +1,3 @@ +delimiter: + to: caoxl78320(a)163.com + bcc: caoxl78320(a)163.com, wfg(a)mail.ustc.edu.cn, ljx.joe(a)qq.com, huming15(a)163.com diff --git a/lib/parse_mail_list.rb b/lib/parse_mail_list.rb new file mode 100644 index 0000000..1a76a1e --- /dev/null +++ b/lib/parse_mail_list.rb @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'yaml' + +def parse_mail_list(type) + content = {} + mail_list_yaml = './mail_list.yaml' + content = YAML.safe_load(File.open(mail_list_yaml)) if FileTest.exists?(mail_list_yaml) + + return content[type] || {} +end -- 2.23.0

1 1

[PATCH v3 compass-ci] lib: parse mail list from yaml by type as to and bcc mail list
by Cao Xueliang 02 Mar '21

02 Mar '21

Yaml file define: delimiter: to: mail1, mail3 bcc: mail2, mail3 The mail_bisect_result calls parse_mail_list('delimiter') to get the delimiter mail list. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 14 +++++++++++--- lib/mail_list.yaml | 3 +++ lib/parse_mail_list.rb | 12 ++++++++++++ 3 files changed, 26 insertions(+), 3 deletions(-) create mode 100644 lib/mail_list.yaml create mode 100644 lib/parse_mail_list.rb diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 104f098..23a8ade 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -6,6 +6,7 @@ require_relative 'git' require_relative 'es_query' require_relative 'constants' require_relative 'mail_client' +require_relative 'parse_mail_list' require_relative 'assign_account_client' # compose and send email for bisect result @@ -22,10 +23,17 @@ class MailBisectResult @git_commit = GitCommit.new(@work_dir, @commit_id) @to = @git_commit.author_email # now send mail to review - @bcc = 'caoxl(a)crystal.ci, caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + @bcc = 'caoxl78320(a)163.com, huming15(a)163.com, wfg(a)mail.ustc.edu.cn' + end + + def parse_mail_info + mail_hash = parse_mail_list('delimiter') + @to = mail_hash['to'] if mail_hash.key?('to') + @bcc = mail_hash['bcc'] if mail_hash.key?('bcc') end def create_send_email + parse_mail_info send_report_mail(compose_mail) send_account_mail rm_work_dir @@ -35,7 +43,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' - bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" + report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" @@ -58,7 +66,7 @@ class MailBisectResult #{pkgbuild_repo_url} #{first_bad_commit_job_url} #{bisect_job_url} - #{bisect_report_doc} + #{report_doc} Regards, Compass CI team BODY diff --git a/lib/mail_list.yaml b/lib/mail_list.yaml new file mode 100644 index 0000000..d982c93 --- /dev/null +++ b/lib/mail_list.yaml @@ -0,0 +1,3 @@ +delimiter: + to: caoxl78320(a)163.com + bcc: caoxl78320(a)163.com, wfg(a)mail.ustc.edu.cn, ljx.joe(a)qq.com, huming15(a)163.com diff --git a/lib/parse_mail_list.rb b/lib/parse_mail_list.rb new file mode 100644 index 0000000..1a76a1e --- /dev/null +++ b/lib/parse_mail_list.rb @@ -0,0 +1,12 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# frozen_string_literal: true + +require 'yaml' + +def parse_mail_list(type) + content = {} + mail_list_yaml = './mail_list.yaml' + content = YAML.safe_load(File.open(mail_list_yaml)) if FileTest.exists?(mail_list_yaml) + + return content[type] || {} +end -- 2.23.0

1 0

[PATCH v2 lkp-tests] daemon: add mysql
by Wei Jihui 02 Mar '21

02 Mar '21

add mysql for sysbench-mysql test Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..89d2318c --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,104 @@ +#!/bin/sh +# - nic +# - device +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh + +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$device" + mount /dev/"$device" /data + + mkdir -p /data/mysql/data /data/mysql/share /data/mysql/tmp /data/mysql/run /data/mysql/log + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +start_mysql_server() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +disable_irqbalance() +{ + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 + pkill -9 irqbalance >/dev/null 2>&1 + systemctl start tuned >/dev/null 2>&1 +} + +intrrupt_nic_binding() +{ + local core_ranges='55-79 93-95' + ethtool -L $nic combined 28 + for core_range in $core_ranges + do + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + done + ethtool -l $nic +} + +net_io_cache_setting() +{ + echo deadline > /sys/block/"$device"/queue/scheduler + echo 2048 > /sys/block/"$device"/queue/nr_requests +} + +getcpu_process_bind_core() +{ + cd /getcpu || exit + insmod getcpu-hi.ko + numactl -C 0-92 -i 0-3 "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + start_mysql_server + create_database + disable_irqbalance + intrrupt_nic_binding + net_io_cache_setting + getcpu_process_bind_core +} +main -- 2.23.0

2 1

[PATCH compass-ci 1/2] sbin/upstream-config: add upstream-config file
by Li Yuanchao 02 Mar '21

02 Mar '21

To support more than one upstream repos warehouse Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/upstream-config | 10 ++++++++++ 1 file changed, 10 insertions(+) create mode 100644 sbin/upstream-config diff --git a/sbin/upstream-config b/sbin/upstream-config new file mode 100644 index 0000000..e3b167a --- /dev/null +++ b/sbin/upstream-config @@ -0,0 +1,10 @@ +--- +upstreams: +- url: https://gitee.com/wu_fengguang/upstream-repos.git + location: upstream +- url: https://gitee.com/wu_fengguang/ansible-repos.git + location: ansible +- url: https://gitee.com/wu_fengguang/dockerfile-repos.git + location: dockerfile +- url: https://gitee.com/wu_fengguang/pkgbuild-repos.git + location: pkgbuild -- 2.23.0

2 3

[PATCH compass-ci 2/2] lib/git_mirror.rb: support more than one upstream repos warehouse
by Li Yuanchao 02 Mar '21

02 Mar '21

Support monitor repos in upstream-repos/ansible-repos/dockerfile-repos/pkgbuild-repos, they are just like upstream-repos. And which to monitor can be configured in file upstream-config. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 55 ++++++++++++++++++++++++++++++++++------------- 1 file changed, 40 insertions(+), 15 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 22c4e07..4652921 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -81,7 +81,7 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "/srv/git/#{fork_info['belong']}/#{fork_info['git_repo']}" mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = git_repo_download(fork_info['url'], mirror_dir) feedback(fork_info['git_repo'], possible_new_refs) @@ -107,6 +107,7 @@ class MirrorMain @defaults = {} @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") + clone_upstream_repo load_fork_info connection_init handle_webhook @@ -124,35 +125,37 @@ class MirrorMain @fork_stat[git_repo] = get_fork_stat(git_repo) end - def load_defaults(repodir) + def load_defaults(repodir, belong) defaults_file = "#{repodir}/DEFAULTS" return unless File.exist?(defaults_file) - defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") + defaults_key = repodir == "#{REPO_DIR}/#{belong}" ? belong : repodir.delete_prefix("#{REPO_DIR}/#{belong}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) - @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key], belong) end - def traverse_repodir(repodir) + def traverse_repodir(repodir, belong) if File.directory? repodir - load_defaults(repodir) + load_defaults(repodir, belong) entry_list = Dir.children(repodir) - ['DEFAULTS'] entry_list.each do |entry| next if entry.start_with? '.' - traverse_repodir("#{repodir}/#{entry}") + traverse_repodir("#{repodir}/#{entry}", belong) end else return if File.dirname(repodir) == REPO_DIR - project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/#{belong}/") fork_name = File.basename(repodir) - load_repo_file(repodir, project, fork_name) + load_repo_file(repodir, project, fork_name, belong) end end def load_fork_info - traverse_repodir(REPO_DIR) + @upstreams['upstreams'].each do |repo| + traverse_repodir("#{REPO_DIR}/#{repo['location']}", repo['location']) + end end def create_workers @@ -221,14 +224,16 @@ end # main thread class MirrorMain - def load_repo_file(repodir, project, fork_name) + def load_repo_file(repodir, project, fork_name, belong) git_repo = "#{project}/#{fork_name}" git_info = YAML.safe_load(File.open(repodir)) return if git_info.nil? || git_info['url'].nil? @git_info[git_repo] = git_info @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) + @git_info[git_repo]['belong'] = belong + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) + fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -453,8 +458,8 @@ end # main thread class MirrorMain - def merge_defaults(object_key, object) - return object if object_key == 'default' + def merge_defaults(object_key, object, belong) + return object if object_key == belong defaults_key = File.dirname(object_key) while defaults_key != '.' @@ -462,8 +467,28 @@ class MirrorMain defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(object) if @defaults['default'] + return @defaults[belong].merge(object) if @defaults[belong] return object end + + def clone_upstream_repo + if File.exist?("#{ENV['CCI_SRC']}/sbin/upstream-config") + @upstreams = YAML.safe_load(File.open('../sbin/upstream-config')) + @upstreams['upstreams'].each do |repo| + url = get_url(repo['url']) + %x(git clone #{url} #{REPO_DIR}/#{repo['location']} 2>&1) + end + else + puts 'ERROR: No upstream-config file' + return -1 + end + end + + def get_url(url) + if url.include?('gitee.com/') && File.exist?("/srv/git/#{url.delete_prefix('https://')}") + url = "/srv/git/#{url.delete_prefix('https://')}" + end + return url + end end -- 2.23.0

2 3

[PATCH v3 compass-ci 2/2] git_mirror.rb: reload more internal files when DEFAULTS change
by Li Yuanchao 02 Mar '21

02 Mar '21

DEFAULTS files are used by many files in the inner directory, so when DEFAULTS files change, those inner files should also be reload. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index bad73d5..bb0c9f9 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -285,10 +285,11 @@ class MirrorMain def reload(file_list, belong) system("git -C #{REPO_DIR}/#{belong} pull") + reload_defaults(file_list, belong) file_list.each_line do |file| - next if File.basename(file) == '.ignore' - file = file.chomp + next if File.basename(file) == '.ignore' || File.basename(file) == 'DEFAULTS' + repo_dir = "#{REPO_DIR}/#{belong}/#{file}" load_repo_file(repo_dir, File.dirname(file), File.basename(file), belong) if File.file?(repo_dir) end @@ -497,4 +498,15 @@ class MirrorMain end return false end + + def reload_defaults(file_list, belong) + file_list.each_line do |file| + file = file.chomp + next unless File.basename(file) == 'DEFAULTS' + + repodir = "#{REPO_DIR}/#{belong}/#{File.dirname(file)}" + load_defaults(repodir, belong) + traverse_repodir(repodir, belong) + end + end end -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] lib/git_mirror.rb: support more than one upstream repos warehouse
by Li Yuanchao 02 Mar '21

02 Mar '21

Support monitor repos in upstream-repos/ansible-repos/dockerfile-repos/pkgbuild-repos, they are just like upstream-repos. And which to monitor can be configured in file upstream-config. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 89 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 60 insertions(+), 29 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 22c4e07..bad73d5 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -81,7 +81,7 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "/srv/git/#{fork_info['belong']}/#{fork_info['git_repo']}" mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = git_repo_download(fork_info['url'], mirror_dir) feedback(fork_info['git_repo'], possible_new_refs) @@ -107,6 +107,7 @@ class MirrorMain @defaults = {} @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") + clone_upstream_repo load_fork_info connection_init handle_webhook @@ -124,35 +125,37 @@ class MirrorMain @fork_stat[git_repo] = get_fork_stat(git_repo) end - def load_defaults(repodir) + def load_defaults(repodir, belong) defaults_file = "#{repodir}/DEFAULTS" return unless File.exist?(defaults_file) - defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") + defaults_key = repodir == "#{REPO_DIR}/#{belong}" ? belong : repodir.delete_prefix("#{REPO_DIR}/#{belong}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) - @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key], belong) end - def traverse_repodir(repodir) + def traverse_repodir(repodir, belong) if File.directory? repodir - load_defaults(repodir) + load_defaults(repodir, belong) entry_list = Dir.children(repodir) - ['DEFAULTS'] entry_list.each do |entry| next if entry.start_with? '.' - traverse_repodir("#{repodir}/#{entry}") + traverse_repodir("#{repodir}/#{entry}", belong) end else return if File.dirname(repodir) == REPO_DIR - project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/#{belong}/") fork_name = File.basename(repodir) - load_repo_file(repodir, project, fork_name) + load_repo_file(repodir, project, fork_name, belong) end end def load_fork_info - traverse_repodir(REPO_DIR) + @upstreams['upstreams'].each do |repo| + traverse_repodir("#{REPO_DIR}/#{repo['location']}", repo['location']) + end end def create_workers @@ -182,7 +185,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' + return reload_fork_info(git_repo) if is_upstream_repo?(git_repo) new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -221,14 +224,16 @@ end # main thread class MirrorMain - def load_repo_file(repodir, project, fork_name) + def load_repo_file(repodir, project, fork_name, belong) git_repo = "#{project}/#{fork_name}" git_info = YAML.safe_load(File.open(repodir)) return if git_info.nil? || git_info['url'].nil? @git_info[git_repo] = git_info @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) + @git_info[git_repo]['belong'] = belong + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) + fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -247,7 +252,7 @@ class MirrorMain def get_cur_refs(git_repo) return if @git_info[git_repo]['is_submodule'] - mirror_dir = "/srv/git/#{git_repo}.git" + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}.git" show_ref_out = %x(git -C #{mirror_dir} show-ref --heads 2>/dev/null) cur_refs = { heads: {} } show_ref_out.each_line do |line| @@ -266,27 +271,26 @@ class MirrorMain return new_refs end - def reload_fork_info - upstream_repos = 'u/upstream-repos/upstream-repos' + def reload_fork_info(upstream_repos) if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] new_refs = check_new_refs(upstream_repos) new_commit = new_refs[:heads]['refs/heads/master'] - changed_files = %x(git -C /srv/git/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) - reload(changed_files) + changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) + reload(changed_files, @git_info[upstream_repos]['belong']) end end - def reload(file_list) - system("git -C #{REPO_DIR} pull") + def reload(file_list, belong) + system("git -C #{REPO_DIR}/#{belong} pull") file_list.each_line do |file| next if File.basename(file) == '.ignore' file = file.chomp - repo_dir = "#{REPO_DIR}/#{file}" - load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) + repo_dir = "#{REPO_DIR}/#{belong}/#{file}" + load_repo_file(repo_dir, File.dirname(file), File.basename(file), belong) if File.file?(repo_dir) end end @@ -357,7 +361,7 @@ class MirrorMain end end - def handle_submodule(submodule) + def handle_submodule(submodule, belong) submodule.each_line do |line| next unless line.include?('url = ') @@ -365,7 +369,7 @@ class MirrorMain git_repo = url.split('://')[1] if url.include?('://') break unless git_repo - @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } + @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -378,11 +382,11 @@ class MirrorMain return true end - mirror_dir = "/srv/git/#{git_repo}.git" + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}.git" submodule = %x(git -C #{mirror_dir} show HEAD:.gitmodules 2>/dev/null) return if submodule.empty? - handle_submodule(submodule) + handle_submodule(submodule, @git_info[git_repo]['belong']) end def get_fork_stat(git_repo) @@ -453,8 +457,8 @@ end # main thread class MirrorMain - def merge_defaults(object_key, object) - return object if object_key == 'default' + def merge_defaults(object_key, object, belong) + return object if object_key == belong defaults_key = File.dirname(object_key) while defaults_key != '.' @@ -462,8 +466,35 @@ class MirrorMain defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(object) if @defaults['default'] + return @defaults[belong].merge(object) if @defaults[belong] return object end + + def clone_upstream_repo + if File.exist?('/etc/compass-ci/defaults/upstream-config') + @upstreams = YAML.safe_load(File.open('/etc/compass-ci/defaults/upstream-config')) + @upstreams['upstreams'].each do |repo| + url = get_url(repo['url']) + %x(git clone #{url} #{REPO_DIR}/#{repo['location']} 2>&1) + end + else + puts 'ERROR: No upstream-config file' + return -1 + end + end + + def get_url(url) + if url.include?('gitee.com/') && File.exist?("/srv/git/#{url.delete_prefix('https://')}") + url = "/srv/git/#{url.delete_prefix('https://')}" + end + return url + end + + def is_upstream_repo?(git_repo) + @upstreams['upstreams'].each do |repo| + return true if git_repo == repo['git_repo'] + end + return false + end end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] modify all SRV_HTTP_HOST/PORT
by Lu Weitao 02 Mar '21

02 Mar '21

[Why] split compass-ci/container/srv-http to 6 service pre dir pre service with a port [How] SRV_HTTP_PORT => SRV_HTTP_RESULT_HOST/PORT, user: compass-ci/lib/mail_job_result.rb compass-ci/container/extract-stats SRV_HTTP_OS_HOST/PORT, user: compass-ci/sbin/download-rootfs SRV_HTTP_GIT_HOST/PORT, user: compass-ci/lib/mail_bisect_result.rb SRV_HTTP_CCI_HOST/PORT, user: compass-ci/lib/compass-ci/src/scheduler/find_job_boot.cr compass-ci/providers/lib/domain.rb add them(HOST/PORT) to: compass-ci/lib/constants.rb compass-ci/sparrow/3-code/dev-env Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 12 ++++++------ container/scheduler/start | 6 +++--- lib/constants.rb | 8 ++++++-- lib/mail_job_result.rb | 2 +- providers/lib/domain.rb | 4 ++-- sbin/download-rootfs | 2 +- sparrow/3-code/dev-env | 24 ++++++++++++++++++------ src/scheduler/constants.cr | 4 ++-- src/scheduler/find_job_boot.cr | 4 ++-- 9 files changed, 41 insertions(+), 25 deletions(-) diff --git a/container/extract-stats/start b/container/extract-stats/start index c158977..5033754 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -14,8 +14,8 @@ names = Set.new %w[ ES_PORT SEND_MAIL_HOST SEND_MAIL_PORT - SRV_HTTP_HOST - SRV_HTTP_PORT + SRV_HTTP_RESULT_HOST + SRV_HTTP_RESULT_PORT lab ] @@ -24,8 +24,8 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' -SRV_HTTP_HOST = defaults['SRV_HTTP_HOST'] || Open3.capture3('curl ip.sb')[0].chomp -SRV_HTTP_PORT = defaults['SRV_HTTP_PORT'] || '11300' +SRV_HTTP_RESULT_HOST = defaults['SRV_HTTP_RESULT_HOST'] || Open3.capture3('curl ip.sb')[0].chomp +SRV_HTTP_RESULT_PORT = defaults['SRV_HTTP_RESULT_PORT'] || '11300' docker_rm "extract-stats" cmd = %w[ @@ -36,8 +36,8 @@ cmd = %w[ ] + env + %W[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} - -e SRV_HTTP_HOST=#{SRV_HTTP_HOST} - -e SRV_HTTP_PORT=#{SRV_HTTP_PORT} + -e SRV_HTTP_RESULT_HOST=#{SRV_HTTP_RESULT_HOST} + -e SRV_HTTP_RESULT_PORT=#{SRV_HTTP_RESULT_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} diff --git a/container/scheduler/start b/container/scheduler/start index 4aaa71d..d83fcc2 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -25,8 +25,8 @@ names = Set.new %w[ SSHR_PORT SSHR_PORT_BASE SSHR_PORT_LEN - SRV_HTTP_HOST - SRV_HTTP_PORT + SRV_HTTP_CCI_HOST + SRV_HTTP_CCI_PORT lab ] @@ -37,7 +37,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' -defaults['SRV_HTTP_PORT'] ||= '11300' +defaults['SRV_HTTP_CCI_PORT'] ||= '11306' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/lib/constants.rb b/lib/constants.rb index 5ed4d57..8d8e488 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -11,8 +11,12 @@ ES_PORT ||= config['ES_PORT'] || 9200 SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 -SRV_HTTP_HOST ||= config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' -SRV_HTTP_PORT ||= config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 +SRV_HTTP_RESULT_HOST ||= config['SRV_HTTP_RESULT_HOST'] || ENV['SRV_HTTP_RESULT_HOST'] || '172.17.0.1' +SRV_HTTP_OS_HOST ||= config['SRV_HTTP_OS_HOST'] || ENV['SRV_HTTP_OS_HOST'] || '172.17.0.1' +SRV_HTTP_GIT_HOST ||= config['SRV_HTTP_GIT_HOST'] || ENV['SRV_HTTP_GIT_HOST'] || '172.17.0.1' +SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 11300 +SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 11304 +SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 11305 SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 796a4f3..e588470 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -9,7 +9,7 @@ require 'json' # compose and send email for job result class MailJobResult - def initialize(job_id, result_host = SRV_HTTP_HOST, result_port = SRV_HTTP_PORT) + def initialize(job_id, result_host = SRV_HTTP_RESULT_HOST, result_port = SRV_HTTP_RESULT_PORT) @job_id = job_id @result_host = result_host @result_port = result_port diff --git a/providers/lib/domain.rb b/providers/lib/domain.rb index 8df9c06..318e68d 100644 --- a/providers/lib/domain.rb +++ b/providers/lib/domain.rb @@ -46,8 +46,8 @@ class Domain end def load_xml_file(filepath) - host = @context.info['SRV_HTTP_HOST'] - port = @context.info['SRV_HTTP_PORT'] + host = @context.info['SRV_HTTP_CCI_HOST'] + port = @context.info['SRV_HTTP_CCI_PORT'] system "wget --timestamping --progress=bar:force http://#{host}:#{port}/cci/libvirt-xml/#{filepath}" %x(basename #{filepath}).chomp end diff --git a/sbin/download-rootfs b/sbin/download-rootfs index e1d4091..74afbf6 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -25,7 +25,7 @@ download_os_rootfs() mkdir -p $rootfs_dir cd $rootfs_dir - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${os}/${os_arch}/${…" + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" wget $download_url [ "$?" -ne 0 ] && { diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index cc94e01..698a4a2 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -16,8 +16,14 @@ server_name=$(hostname | cut -f1 -d.) : ${OS_HTTP_HOST:=$server_ip} : ${SEND_MAIL_HOST:=$server_ip} : ${SEND_MAIL_PORT:=49000} -: ${SRV_HTTP_HOST:=$server_ip} -: ${SRV_HTTP_PORT:=11300} +: ${SRV_HTTP_RESULT_HOST:=$server_ip} +: ${SRV_HTTP_OS_HOST:=$server_ip} +: ${SRV_HTTP_GIT_HOST:=$server_ip} +: ${SRV_HTTP_CCI_HOST:=$server_ip} +: ${SRV_HTTP_RESULT_PORT:=11300} +: ${SRV_HTTP_OS_PORT:=11304} +: ${SRV_HTTP_GIT_PORT:=11305} +: ${SRV_HTTP_CCI_PORT:=11306} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} : ${INITRD_HTTP_HOST:=$server_ip} @@ -49,8 +55,14 @@ GIT_SERVER: $GIT_SERVER OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port -SRV_HTTP_HOST: $SRV_HTTP_HOST -SRV_HTTP_PORT: $SRV_HTTP_PORT +SRV_HTTP_RESULT_HOST: $SRV_HTTP_RESULT_HOST +SRV_HTTP_OS_HOST: $SRV_HTTP_OS_HOST +SRV_HTTP_GIT_HOST: $SRV_HTTP_GIT_HOST +SRV_HTTP_CCI_HOST: $SRV_HTTP_CCI_HOST +SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT +SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT +SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT +SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host SEND_MAIL_HOST: $SEND_MAIL_HOST SEND_MAIL_PORT: $SEND_MAIL_PORT @@ -77,8 +89,8 @@ LKP_SERVER: GIT_SERVER: GITCACHE_HOST: GITCACHE_PORT: -SRV_HTTP_HOST: -SRV_HTTP_PORT: +SRV_HTTP_CCI_HOST: +SRV_HTTP_CCI_PORT: SEND_MAIL_HOST: SEND_MAIL_PORT: INITRD_HTTP_HOST: diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 668da6d..d7d7a9a 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -23,8 +23,8 @@ INITRD_HTTP_PORT = (ENV.has_key?("INITRD_HTTP_PORT") ? ENV["INITRD_HTTP_PORT"] : OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1") OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 -SRV_HTTP_HOST = (ENV.has_key?("SRV_HTTP_HOST") ? ENV["SRV_HTTP_HOST"] : "172.17.0.1") -SRV_HTTP_PORT = (ENV.has_key?("SRV_HTTP_PORT") ? ENV["SRV_HTTP_PORT"] : "11300") +SRV_HTTP_CCI_HOST = (ENV.has_key?("SRV_HTTP_CCI_HOST") ? ENV["SRV_HTTP_CCI_HOST"] : "172.17.0.1") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11306") CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 8dd22a8..3323d4d 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -191,8 +191,8 @@ class Sched "LKP_SERVER" => job["LKP_SERVER"]?, "vt" => _vt, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", - "SRV_HTTP_HOST" => SRV_HTTP_HOST, - "SRV_HTTP_PORT" => SRV_HTTP_PORT, + "SRV_HTTP_CCI_HOST" => SRV_HTTP_CCI_HOST, + "SRV_HTTP_CCI_PORT" => SRV_HTTP_CCI_PORT, }.to_json end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] container/srv-http: split up service
by Lu Weitao 02 Mar '21

02 Mar '21

[Why] This service ties up different dirs to the same service. However we'll need the flexibility to hold the dirs in different file servers. [How] one service per dir, and access-like: result: http://$host:11300/ pub: http://$host:11302/ initrd: http://$host:11303/ os: http://$host:11304/ git: http://$host:11305/ cci: http://$host:11306/ Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/srv-http/start | 27 ++++++--------------------- container/srv-http/start-cci | 19 +++++++++++++++++++ container/srv-http/start-git | 19 +++++++++++++++++++ container/srv-http/start-initrd | 20 ++++++++++++++++++++ container/srv-http/start-os | 19 +++++++++++++++++++ container/srv-http/start-pub | 19 +++++++++++++++++++ container/srv-http/start-result | 19 +++++++++++++++++++ 7 files changed, 121 insertions(+), 21 deletions(-) create mode 100755 container/srv-http/start-cci create mode 100755 container/srv-http/start-git create mode 100755 container/srv-http/start-initrd create mode 100755 container/srv-http/start-os create mode 100755 container/srv-http/start-pub create mode 100755 container/srv-http/start-result diff --git a/container/srv-http/start b/container/srv-http/start index fa56fca..f262d81 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -2,24 +2,9 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -. $CCI_SRC/container/defconfig.sh -docker_rm srv-http - -cmd=( - docker run - --restart=always - --name srv-http - -p 11300:11300 - -v /srv/pub:/srv/pub:ro - -v /etc/localtime:/etc/localtime:ro - -v /srv/result:/srv/result:ro - -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro - -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro - -v /srv/os:/srv/os:ro - -v /srv/git/archlinux:/srv/git/archlinux:ro - -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro - -d - srv-http -) - -"${cmd[@]}" +./start-pub +./start-result +./start-initrd +./start-os +./start-git +./start-cci diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci new file mode 100755 index 0000000..78ab149 --- /dev/null +++ b/container/srv-http/start-cci @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-cci + +cmd=( + docker run + --restart=always + --name srv-http-cci + -p 11306:11300 + -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-git b/container/srv-http/start-git new file mode 100755 index 0000000..f2292bf --- /dev/null +++ b/container/srv-http/start-git @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-git + +cmd=( + docker run + --restart=always + --name srv-http-git + -p 11305:11300 + -v /srv/git/archlinux:/srv/git/archlinux:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd new file mode 100755 index 0000000..7c73ea1 --- /dev/null +++ b/container/srv-http/start-initrd @@ -0,0 +1,20 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-initrd + +cmd=( + docker run + --restart=always + --name srv-http-initrd + -p 11303:11300 + -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro + -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-os b/container/srv-http/start-os new file mode 100755 index 0000000..bb344cf --- /dev/null +++ b/container/srv-http/start-os @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-os + +cmd=( + docker run + --restart=always + --name srv-http-os + -p 11304:11300 + -v /srv/os:/srv/os:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub new file mode 100755 index 0000000..dc1864c --- /dev/null +++ b/container/srv-http/start-pub @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-pub + +cmd=( + docker run + --restart=always + --name srv-http-pub + -p 11302:11300 + -v /srv/pub:/srv/pub:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" diff --git a/container/srv-http/start-result b/container/srv-http/start-result new file mode 100755 index 0000000..f9ecc97 --- /dev/null +++ b/container/srv-http/start-result @@ -0,0 +1,19 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh +docker_rm srv-http-result + +cmd=( + docker run + --restart=always + --name srv-http-result + -p 11300:11300 + -v /srv/result:/srv/result:ro + -v /etc/localtime:/etc/localtime:ro + -d + srv-http +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] sbin/compare won't remove job if job['stats'] no metric start with job['suite']
by Lu Weitao 02 Mar '21

02 Mar '21

[Why] before convert job_list => matrix we remove job from job_list unless: job['stats'].any? { |metric| metric.start_with?(job['suite'])} but some job(such as build-pkg) have no metric start_with suite(build-pkg.xxx) so this logic is unreasonable, we should keep the job in job_list Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 2082c5d..37cf03a 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -50,7 +50,7 @@ def job_is_useful?(job) return unless stats suite = job['_source']['suite'] - return unless suite && stats.keys.any? { |stat| stat.start_with?(suite) } + return unless suite true end -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] git_mirror.rb: reload more internal files when DEFAULTS change
by Li Yuanchao 02 Mar '21

02 Mar '21

DEFAULTS files are used by many files in the inner directory, so when DEFAULTS files change, those inner files should also be reload. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 6417b52..017153f 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -285,10 +285,11 @@ class MirrorMain def reload(file_list, belong) system("git -C #{REPO_DIR}/#{belong} pull") + reload_defaults(file_list, belong) file_list.each_line do |file| - next if File.basename(file) == '.ignore' - file = file.chomp + next if File.basename(file) == '.ignore' || File.basename(file) == 'DEFAULTS' + repo_dir = "#{REPO_DIR}/#{belong}/#{file}" load_repo_file(repo_dir, File.dirname(file), File.basename(file), belong) if File.file?(repo_dir) end @@ -497,4 +498,15 @@ class MirrorMain end return false end + + def reload_defaults(file_list, belong) + file_list.each_line do |file| + file = file.chomp + next unless File.basename(file) == 'DEFAULTS' + + repodir = "#{REPO_DIR}/#{belong}/#{File.dirname(file)}" + load_defaults(repodir, belong) + traverse_repodir(repodir, belong) + end + end end -- 2.23.0

2 2

[PATCH v3 compass-ci 1/2] sched: response ipxe_response value if ipxe_response field in job yaml
by Cao Xueliang 02 Mar '21

02 Mar '21

The install-iso job need scheduler response the ipxe_response value, when consume the install-iso job. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..88db69e 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["suite"] == "install-iso" && job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 3

[PATCH compass-ci] modify all SRV_HTTP_PORT
by Lu Weitao 02 Mar '21

02 Mar '21

[Why] split compass-ci/container/srv-http to 6 service pre dir pre service with a port [How] SRV_HTTP_PORT => SRV_HTTP_RESULT_PORT, user: compass-ci/lib/mail_job_result.rb compass-ci/container/extract-stats SRV_HTTP_OS_PORT, user: compass-ci/sbin/download-rootfs SRV_HTTP_GIT_PORT, user: compass-ci/lib/mail_bisect_result.rb SRV_HTTP_CCI_PORT, user: compass-ci/lib/compass-ci/src/scheduler/find_job_boot.cr compass-ci/providers/lib/domain.rb add them(PORT) to: compass-ci/lib/constants.rb compass-ci/sparrow/3-code/dev-env Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/extract-stats/start | 6 +++--- container/scheduler/start | 4 ++-- lib/constants.rb | 4 +++- lib/mail_job_result.rb | 2 +- providers/lib/domain.rb | 2 +- sbin/download-rootfs | 2 +- sparrow/3-code/dev-env | 12 +++++++++--- src/scheduler/constants.cr | 2 +- src/scheduler/find_job_boot.cr | 2 +- 9 files changed, 22 insertions(+), 14 deletions(-) diff --git a/container/extract-stats/start b/container/extract-stats/start index c158977..ba259af 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -15,7 +15,7 @@ names = Set.new %w[ SEND_MAIL_HOST SEND_MAIL_PORT SRV_HTTP_HOST - SRV_HTTP_PORT + SRV_HTTP_RESULT_PORT lab ] @@ -25,7 +25,7 @@ env = docker_env(defaults) DEFAULT_LKP = '/c/lkp-tests' DEFAULT_CCI = '/c/compass-ci' SRV_HTTP_HOST = defaults['SRV_HTTP_HOST'] || Open3.capture3('curl ip.sb')[0].chomp -SRV_HTTP_PORT = defaults['SRV_HTTP_PORT'] || '11300' +SRV_HTTP_RESULT_PORT = defaults['SRV_HTTP_RESULT_PORT'] || '11300' docker_rm "extract-stats" cmd = %w[ @@ -37,7 +37,7 @@ cmd = %w[ -e LKP_SRC=#{DEFAULT_LKP} -e CCI_SRC=#{DEFAULT_CCI} -e SRV_HTTP_HOST=#{SRV_HTTP_HOST} - -e SRV_HTTP_PORT=#{SRV_HTTP_PORT} + -e SRV_HTTP_RESULT_PORT=#{SRV_HTTP_RESULT_PORT} -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} -v /etc/localtime:/etc/localtime:ro -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} diff --git a/container/scheduler/start b/container/scheduler/start index 4aaa71d..d218626 100755 --- a/container/scheduler/start +++ b/container/scheduler/start @@ -26,7 +26,7 @@ names = Set.new %w[ SSHR_PORT_BASE SSHR_PORT_LEN SRV_HTTP_HOST - SRV_HTTP_PORT + SRV_HTTP_CCI_PORT lab ] @@ -37,7 +37,7 @@ defaults['SSHR_PORT_LEN'] ||= 2000 defaults['SCHED_PORT'] ||= '3000' defaults['SCHED_HOST'] ||= '172.17.0.1' defaults['MASTER_FLUENTD_PORT'] ||= '24224' -defaults['SRV_HTTP_PORT'] ||= '11300' +defaults['SRV_HTTP_CCI_PORT'] ||= '11306' env = docker_env(defaults) CCI_REPOS = ENV['CCI_REPOS'] || '/c' diff --git a/lib/constants.rb b/lib/constants.rb index 5ed4d57..431c8d6 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -12,7 +12,9 @@ SEND_MAIL_HOST ||= config['SEND_MAIL_HOST'] || '172.17.0.1' SEND_MAIL_PORT ||= config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST ||= config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' -SRV_HTTP_PORT ||= config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 +SRV_HTTP_RESULT_PORT ||= config['SRV_HTTP_RESULT_PORT'] || ENV['SRV_HTTP_RESULT_PORT'] || 11300 +SRV_HTTP_OS_PORT ||= config['SRV_HTTP_OS_PORT'] || ENV['SRV_HTTP_OS_PORT'] || 11304 +SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || 11305 SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' diff --git a/lib/mail_job_result.rb b/lib/mail_job_result.rb index 796a4f3..e93c6de 100644 --- a/lib/mail_job_result.rb +++ b/lib/mail_job_result.rb @@ -9,7 +9,7 @@ require 'json' # compose and send email for job result class MailJobResult - def initialize(job_id, result_host = SRV_HTTP_HOST, result_port = SRV_HTTP_PORT) + def initialize(job_id, result_host = SRV_HTTP_HOST, result_port = SRV_HTTP_RESULT_PORT) @job_id = job_id @result_host = result_host @result_port = result_port diff --git a/providers/lib/domain.rb b/providers/lib/domain.rb index 8df9c06..b3de90a 100644 --- a/providers/lib/domain.rb +++ b/providers/lib/domain.rb @@ -47,7 +47,7 @@ class Domain def load_xml_file(filepath) host = @context.info['SRV_HTTP_HOST'] - port = @context.info['SRV_HTTP_PORT'] + port = @context.info['SRV_HTTP_CCI_PORT'] system "wget --timestamping --progress=bar:force http://#{host}:#{port}/cci/libvirt-xml/#{filepath}" %x(basename #{filepath}).chomp end diff --git a/sbin/download-rootfs b/sbin/download-rootfs index e1d4091..74afbf6 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -25,7 +25,7 @@ download_os_rootfs() mkdir -p $rootfs_dir cd $rootfs_dir - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_PORT}/os/${os}/${os_arch}/${…" + download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" wget $download_url [ "$?" -ne 0 ] && { diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index cc94e01..4e6a4cd 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -17,7 +17,10 @@ server_name=$(hostname | cut -f1 -d.) : ${SEND_MAIL_HOST:=$server_ip} : ${SEND_MAIL_PORT:=49000} : ${SRV_HTTP_HOST:=$server_ip} -: ${SRV_HTTP_PORT:=11300} +: ${SRV_HTTP_RESULT_PORT:=11300} +: ${SRV_HTTP_OS_PORT:=11304} +: ${SRV_HTTP_GIT_PORT:=11305} +: ${SRV_HTTP_CCI_PORT:=11306} : ${LOGGING_ES_HOST:=$server_ip} : ${LOGGING_ES_PORT:=9202} : ${INITRD_HTTP_HOST:=$server_ip} @@ -50,7 +53,10 @@ OS_HTTP_HOST: $OS_HTTP_HOST GITCACHE_HOST: $gitcache_host GITCACHE_PORT: $gitcache_port SRV_HTTP_HOST: $SRV_HTTP_HOST -SRV_HTTP_PORT: $SRV_HTTP_PORT +SRV_HTTP_RESULT_PORT: $SRV_HTTP_RESULT_PORT +SRV_HTTP_OS_PORT: $SRV_HTTP_OS_PORT +SRV_HTTP_GIT_PORT: $SRV_HTTP_GIT_PORT +SRV_HTTP_CCI_PORT: $SRV_HTTP_CCI_PORT TASKQUEUE_HOST: $sched_host SEND_MAIL_HOST: $SEND_MAIL_HOST SEND_MAIL_PORT: $SEND_MAIL_PORT @@ -78,7 +84,7 @@ GIT_SERVER: GITCACHE_HOST: GITCACHE_PORT: SRV_HTTP_HOST: -SRV_HTTP_PORT: +SRV_HTTP_CCI_PORT: SEND_MAIL_HOST: SEND_MAIL_PORT: INITRD_HTTP_HOST: diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 668da6d..2cb4588 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -24,7 +24,7 @@ OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1 OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_HTTP_HOST = (ENV.has_key?("SRV_HTTP_HOST") ? ENV["SRV_HTTP_HOST"] : "172.17.0.1") -SRV_HTTP_PORT = (ENV.has_key?("SRV_HTTP_PORT") ? ENV["SRV_HTTP_PORT"] : "11300") +SRV_HTTP_CCI_PORT = (ENV.has_key?("SRV_HTTP_CCI_PORT") ? ENV["SRV_HTTP_CCI_PORT"] : "11306") CCI_REPOS = (ENV.has_key?("CCI_REPOS") ? ENV["CCI_REPOS"] : "/c") LAB_REPO = "lab-z9" diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 8dd22a8..9d46d72 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -192,7 +192,7 @@ class Sched "vt" => _vt, "RESULT_WEBDAV_PORT" => job["RESULT_WEBDAV_PORT"]? || "3080", "SRV_HTTP_HOST" => SRV_HTTP_HOST, - "SRV_HTTP_PORT" => SRV_HTTP_PORT, + "SRV_HTTP_CCI_PORT" => SRV_HTTP_CCI_PORT, }.to_json end -- 2.23.0

2 2

[PATCH v2 compass-ci 1/2] container/dracut-initrd: add module 90lkp/module-setup.sh
by Xu Xijian 01 Mar '21

01 Mar '21

90lkp module a module to deploy lkp module into rootfs. module-setup.sh is a base setup script for every kernel module. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../modules.d/90lkp/module-setup.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/dracut-initrd/modules.d/90lkp/module-setup.sh diff --git a/container/dracut-initrd/modules.d/90lkp/module-setup.sh b/container/dracut-initrd/modules.d/90lkp/module-setup.sh new file mode 100755 index 0000000..c188600 --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/module-setup.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +check() { + return 0 +} + +depends() { + # We do not depend on any modules - just some root + return 0 +} + +# called by dracut +installkernel() { + return 0 +} + +install() { + inst_hook pre-pivot 10 "$moddir/lkp-deploy.sh" +} -- 2.23.0

1 0

[PATCH compass-ci] container/dracut-initrd: add module 90lkp/module-setup.sh
by Xu Xijian 01 Mar '21

01 Mar '21

90lkp module a module to deploy lkp module into rootfs. module-setup.sh is a base setup script for every kernel module. Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- .../modules.d/90lkp/module-setup.sh | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 container/dracut-initrd/modules.d/90lkp/module-setup.sh diff --git a/container/dracut-initrd/modules.d/90lkp/module-setup.sh b/container/dracut-initrd/modules.d/90lkp/module-setup.sh new file mode 100755 index 0000000..c188600 --- /dev/null +++ b/container/dracut-initrd/modules.d/90lkp/module-setup.sh @@ -0,0 +1,19 @@ +#!/bin/bash + +check() { + return 0 +} + +depends() { + # We do not depend on any modules - just some root + return 0 +} + +# called by dracut +installkernel() { + return 0 +} + +install() { + inst_hook pre-pivot 10 "$moddir/lkp-deploy.sh" +} -- 2.23.0

1 0

[PATCH v2 lkp-tests] daemon: add mysql
by Wei Jihui 01 Mar '21

01 Mar '21

add mysql for sysbench-mysql test Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- daemon/mysql | 104 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 104 insertions(+) create mode 100755 daemon/mysql diff --git a/daemon/mysql b/daemon/mysql new file mode 100755 index 00000000..cf498cf7 --- /dev/null +++ b/daemon/mysql @@ -0,0 +1,104 @@ +#!/bin/sh +# - nic +# - device +# mysql_user +# mysql_host +# mysql_port +# mysql_db +# mysql_password +. $LKP_SRC/lib/common.sh +set -x +: "${mysql_user:=root}" +: "${mysql_host:=$server}" +: "${mysql_port:=3306}" +: "${mysql_db:=sysbench_1}" +: "${mysql_password:=mysql_password}" + +create_basedir() +{ + mkdir -p /data + mkfs.xfs -f /dev/"$device" + mount /dev/"$device" /data + + mkdir -p /data/mysql/data /data/mysql/share /data/mysql/tmp /data/mysql/run /data/mysql/log + + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log +} + +start_mysql_server() +{ + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "$mysql is not installed!" >&2 && exit 1 + fi + + "$mysqld" --user=mysql --basedir=/data/mysql --datadir=/data/mysql/data/ --initialize --port="$mysql_port" + sleep 10 + + "$mysql_server" start + sleep 10 + + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 +} + +create_database() +{ + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql_server" stop + sleep 10 +} + +disable_irqbalance() +{ + systemctl stop firewalld >/dev/null 2>&1 + iptables -F >/dev/null 2>&1 + pkill -9 irqbalance >/dev/null 2>&1 + systemctl start tuned >/dev/null 2>&1 +} + +intrrupt_nic_binding() +{ + local core_ranges='55-79 93-95' + ethtool -L $nic combined 28 + for core_range in $core_ranges + do + $LKP_SRC/bin/set_nic_irq_affinity $core_range $nic + done + ethtool -l $nic +} + +net_io_cache_setting() +{ + echo deadline > /sys/block/"$device"/queue/scheduler + echo 2048 > /sys/block/"$device"/queue/nr_requests +} + +getcpu_process_bind_core() +{ + cd /getcpu || exit + insmod getcpu-hi.ko + numactl -C 0-92 -i 0-3 "$mysqld" --defaults-file=/etc/my.cnf --port="$mysql_port" & + sleep 20 +} + +main() +{ + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" + + create_basedir + start_mysql_server + create_database + disable_irqbalance + intrrupt_nic_binding + net_io_cache_setting + getcpu_process_bind_core +} +main -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/git_mirror.rb: support more than one upstream repos warehouse
by Li Yuanchao 01 Mar '21

01 Mar '21

Support monitor repos in upstream-repos/ansible-repos/dockerfile-repos/pkgbuild-repos, they are just like upstream-repos. And which to monitor can be configured in file upstream-config. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 89 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 60 insertions(+), 29 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 22c4e07..6417b52 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -81,7 +81,7 @@ class GitMirror def mirror_sync fork_info = @queue.pop - mirror_dir = "/srv/git/#{fork_info['git_repo']}" + mirror_dir = "/srv/git/#{fork_info['belong']}/#{fork_info['git_repo']}" mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] possible_new_refs = git_repo_download(fork_info['url'], mirror_dir) feedback(fork_info['git_repo'], possible_new_refs) @@ -107,6 +107,7 @@ class MirrorMain @defaults = {} @git_queue = Queue.new @es_client = Elasticsearch::Client.new(url: "http://#{ES_HOST}:#{ES_PORT}") + clone_upstream_repo load_fork_info connection_init handle_webhook @@ -124,35 +125,37 @@ class MirrorMain @fork_stat[git_repo] = get_fork_stat(git_repo) end - def load_defaults(repodir) + def load_defaults(repodir, belong) defaults_file = "#{repodir}/DEFAULTS" return unless File.exist?(defaults_file) - defaults_key = repodir == REPO_DIR ? 'default' : repodir.delete_prefix("#{REPO_DIR}/") + defaults_key = repodir == "#{REPO_DIR}/#{belong}" ? belong : repodir.delete_prefix("#{REPO_DIR}/#{belong}/") @defaults[defaults_key] = YAML.safe_load(File.open(defaults_file)) - @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key]) + @defaults[defaults_key] = merge_defaults(defaults_key, @defaults[defaults_key], belong) end - def traverse_repodir(repodir) + def traverse_repodir(repodir, belong) if File.directory? repodir - load_defaults(repodir) + load_defaults(repodir, belong) entry_list = Dir.children(repodir) - ['DEFAULTS'] entry_list.each do |entry| next if entry.start_with? '.' - traverse_repodir("#{repodir}/#{entry}") + traverse_repodir("#{repodir}/#{entry}", belong) end else return if File.dirname(repodir) == REPO_DIR - project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/") + project = File.dirname(repodir).delete_prefix("#{REPO_DIR}/#{belong}/") fork_name = File.basename(repodir) - load_repo_file(repodir, project, fork_name) + load_repo_file(repodir, project, fork_name, belong) end end def load_fork_info - traverse_repodir(REPO_DIR) + @upstreams['upstreams'].each do |repo| + traverse_repodir("#{REPO_DIR}/#{repo['location']}", repo['location']) + end end def create_workers @@ -182,7 +185,7 @@ class MirrorMain update_fork_stat(git_repo, feedback_info[:possible_new_refs]) return unless feedback_info[:possible_new_refs] - return reload_fork_info if git_repo == 'u/upstream-repos/upstream-repos' + return reload_fork_info(git_repo) if is_upstream_repo?(git_repo) new_refs = check_new_refs(git_repo) return if new_refs[:heads].empty? @@ -221,14 +224,16 @@ end # main thread class MirrorMain - def load_repo_file(repodir, project, fork_name) + def load_repo_file(repodir, project, fork_name, belong) git_repo = "#{project}/#{fork_name}" git_info = YAML.safe_load(File.open(repodir)) return if git_info.nil? || git_info['url'].nil? @git_info[git_repo] = git_info @git_info[git_repo]['git_repo'] = git_repo - @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo]) + @git_info[git_repo]['belong'] = belong + @git_info[git_repo] = merge_defaults(git_repo, @git_info[git_repo], belong) + fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -247,7 +252,7 @@ class MirrorMain def get_cur_refs(git_repo) return if @git_info[git_repo]['is_submodule'] - mirror_dir = "/srv/git/#{git_repo}.git" + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}.git" show_ref_out = %x(git -C #{mirror_dir} show-ref --heads 2>/dev/null) cur_refs = { heads: {} } show_ref_out.each_line do |line| @@ -266,27 +271,26 @@ class MirrorMain return new_refs end - def reload_fork_info - upstream_repos = 'u/upstream-repos/upstream-repos' + def reload_fork_info(upstream_repos) if @git_info[upstream_repos][:cur_refs].empty? @git_info[upstream_repos][:cur_refs] = get_cur_refs(upstream_repos) else old_commit = @git_info[upstream_repos][:cur_refs][:heads]['refs/heads/master'] new_refs = check_new_refs(upstream_repos) new_commit = new_refs[:heads]['refs/heads/master'] - changed_files = %x(git -C /srv/git/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) - reload(changed_files) + changed_files = %x(git -C /srv/git/#{@git_info[upstream_repos]['belong']}/#{upstream_repos}.git diff --name-only #{old_commit}...#{new_commit}) + reload(changed_files, @git_info[upstream_repos]['belong']) end end - def reload(file_list) - system("git -C #{REPO_DIR} pull") + def reload(file_list, belong) + system("git -C #{REPO_DIR}/#{belong} pull") file_list.each_line do |file| next if File.basename(file) == '.ignore' file = file.chomp - repo_dir = "#{REPO_DIR}/#{file}" - load_repo_file(repo_dir, File.dirname(file), File.basename(file)) if File.file?(repo_dir) + repo_dir = "#{REPO_DIR}/#{belong}/#{file}" + load_repo_file(repo_dir, File.dirname(file), File.basename(file), belong) if File.file?(repo_dir) end end @@ -357,7 +361,7 @@ class MirrorMain end end - def handle_submodule(submodule) + def handle_submodule(submodule, belong) submodule.each_line do |line| next unless line.include?('url = ') @@ -365,7 +369,7 @@ class MirrorMain git_repo = url.split('://')[1] if url.include?('://') break unless git_repo - @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true } + @git_info[git_repo] = { 'url' => url, 'git_repo' => git_repo, 'is_submodule' => true, 'belong' => belong } fork_stat_init(git_repo) @priority_queue.push git_repo, @priority @priority += 1 @@ -378,11 +382,11 @@ class MirrorMain return true end - mirror_dir = "/srv/git/#{git_repo}.git" + mirror_dir = "/srv/git/#{@git_info[git_repo]['belong']}/#{git_repo}.git" submodule = %x(git -C #{mirror_dir} show HEAD:.gitmodules 2>/dev/null) return if submodule.empty? - handle_submodule(submodule) + handle_submodule(submodule, @git_info[git_repo]['belong']) end def get_fork_stat(git_repo) @@ -453,8 +457,8 @@ end # main thread class MirrorMain - def merge_defaults(object_key, object) - return object if object_key == 'default' + def merge_defaults(object_key, object, belong) + return object if object_key == belong defaults_key = File.dirname(object_key) while defaults_key != '.' @@ -462,8 +466,35 @@ class MirrorMain defaults_key = File.dirname(defaults_key) end - return @defaults['default'].merge(object) if @defaults['default'] + return @defaults[belong].merge(object) if @defaults[belong] return object end + + def clone_upstream_repo + if File.exist?("#{ENV['CCI_SRC']}/sbin/upstream-config") + @upstreams = YAML.safe_load(File.open("#{ENV['CCI_SRC']}/sbin/upstream-config")) + @upstreams['upstreams'].each do |repo| + url = get_url(repo['url']) + %x(git clone #{url} #{REPO_DIR}/#{repo['location']} 2>&1) + end + else + puts 'ERROR: No upstream-config file' + return -1 + end + end + + def get_url(url) + if url.include?('gitee.com/') && File.exist?("/srv/git/#{url.delete_prefix('https://')}") + url = "/srv/git/#{url.delete_prefix('https://')}" + end + return url + end + + def is_upstream_repo?(git_repo) + @upstreams['upstreams'].each do |repo| + return true if git_repo == repo['git_repo'] + end + return false + end end -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sparrow/4-docker/buildall: add $action value "start-only"
by Liu Yinsi 01 Mar '21

01 Mar '21

[usage] ./buildall start-only [how] before: ./buildall run-only: call build_depends(), check images exist or not, then call start_depends() after: ./buildall start-only: only call start_depends() [why] for some users locally deploy compass-ci, they will not build images, and some users can't install lockfile-create command, only run containers, no need to check images exist or not, so only call start_depends(). Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 8d08603..a7ace65 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -70,7 +70,7 @@ tmpdir=$(mktemp -d) for dir in $CONTAINER_PATH/*/ do - build_depends $dir + [ "$action" != "start-only" ] && build_depends $dir start_depends $dir done -- 2.23.0

1 0

[PATCH v4 compass-ci 1/2] sparrow/4-docker/buildall: check whether skip ssh-r
by Liu Yinsi 01 Mar '21

01 Mar '21

[why] for user locally deploy compass-ci, ssh-r is not exists, so check when ssh-r already exists, skip ssh-r. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index bb77a1f..8d08603 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,10 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && exit + [ "$container_name" == "ssh-r" ] && { + container_id=$(docker ps -aqf name="ssh_r") + [ -n "$container_id" ] && exit + } [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/2] pkg/mysql: delete mysql specified port
by Wei Jihui 01 Mar '21

01 Mar '21

Can not specified port in the my.cnf, user may need to configure it flexibly. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- pkg/mysql/PKGBUILD | 2 +- pkg/mysql/my.cnf | 1 - 2 files changed, 1 insertion(+), 2 deletions(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 0f7d8bc8..304f6a4a 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -9,7 +9,7 @@ source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '7bb0372ec4e9281813a9afc67c6cfc20') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '1390631a1ea1832f9c27d07fcc1e536e') prepare() { diff --git a/pkg/mysql/my.cnf b/pkg/mysql/my.cnf index 9732af40..d95811dd 100644 --- a/pkg/mysql/my.cnf +++ b/pkg/mysql/my.cnf @@ -13,7 +13,6 @@ socket=/data/mysql/run/mysql.sock tmpdir=/data/mysql/tmp datadir=/data/mysql/data default_authentication_plugin=mysql_native_password -port=3309 user=mysql memlock=on bind-address=0.0.0.0 -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] lib/job2sh.rb: fix tests script args can not be upper letter key
by Wei Jihui 01 Mar '21

01 Mar '21

[input] testcase: ab: 1 AB: 2 [error ouput at job.sh] run_test ab=1 _65__66_=2 ... [how] pry(main)> key="AB" => "AB" pry(main)> key.gsub(/[^a-z0-9_]/) { |m| '_' + m.getbyte(0).to_s +'_' } => "_65__66_" pry(main)> key.gsub(/[^a-zA-Z0-9_]/) { |m| '_' + m.getbyte(0).to_s+ '_' } => "AB" Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job2sh.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 82ec7d24..e058483d 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -20,7 +20,7 @@ def valid_shell_variable?(key) end def shell_encode_keyword(key) - key.gsub(/[^a-z0-9_]/) { |m| '_' + m.getbyte(0).to_s + '_' } + key.gsub(/[^a-zA-Z0-9_]/) { |m| '_' + m.getbyte(0).to_s + '_' } end def shell_escape_expand(val) -- 2.23.0

1 0

[PATCH lkp-tests] lib/job2sh.rb: fix tests script args can not be upper letter key
by Wei Jihui 01 Mar '21

01 Mar '21

[input] testcase: ab: 1 AB: 2 [error ouput at job.sh] run_test ab=1 _65__66_=2 ... [how] pry(main)> key="AB" => "AB" pry(main)> key.gsub(/[^a-z0-9_]/) { |m| '_' + m.getbyte(0).to_s +'_' } => "_65__66_" pry(main)> key.gsub(/[^a-zA-Z0-9_]/) { |m| '_' + m.getbyte(0).to_s+ '_' } => "AB" Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job2sh.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 82ec7d24..e058483d 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -20,7 +20,7 @@ def valid_shell_variable?(key) end def shell_encode_keyword(key) - key.gsub(/[^a-z0-9_]/) { |m| '_' + m.getbyte(0).to_s + '_' } + key.gsub(/[^a-zA-Z0-9_]/) { |m| '_' + m.getbyte(0).to_s + '_' } end def shell_escape_expand(val) -- 2.23.0

1 0