- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] hosts: add dc-16g
by Cao Xueliang 09 Feb '21

09 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- hosts/dc-16g | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 hosts/dc-16g diff --git a/hosts/dc-16g b/hosts/dc-16g new file mode 100644 index 000000000..1f16373e6 --- /dev/null +++ b/hosts/dc-16g @@ -0,0 +1,3 @@ +nr_cpu: 16 +memory: 16g + -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/es-find: support count keyword
by Lu Weitao 09 Feb '21

09 Feb '21

[Example] es-find tbox_group=vm-2p8g -c/--count suite,job_state { "iperf": { "finished": 50, "failed": 40, "incomplete": 4, "boot": 3 }, "iperf-walk-os-test": { "finished": 167, "failed": 22 }, "borrow": { "finished": 47, "incomplete": 17, "failed": 2 }, ... } Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/sbin/es-find b/sbin/es-find index ed54955..2c3b3dc 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -6,6 +6,7 @@ # Usage: # es-find key=val ... # es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." +# es-find tbox_group=vm-2p8g -c suite,job_state # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -74,6 +75,7 @@ end options = { nr_jobs: 10, desc_keyword: 'start_time' } range = nil +count = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -97,6 +99,10 @@ opt_parser = OptionParser.new do |opts| range = r end + opts.on('-c', '--count KEYWORD', 'count keyword: suite|job_state|testbox|id|...') do |c| + count = c + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -108,6 +114,11 @@ items = parse_argv items[:range] = prase_range(range) if range es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +if count + count_keywords = count.split(',') + query_result = es.query_fields(count_keywords, items) +else + query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) +end json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: fix PKGBUILD url 404
by Cao Xueliang 09 Feb '21

09 Feb '21

[error] http://api.compass-ci.openeuler.org:11300/archlinux/aur-v/vifm-git 404 [how] Need the add git to the url. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 5e1103c..d43785b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -36,7 +36,7 @@ class MailBisectResult prefix_srv = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_PORT}" bisect_job_url = ENV['result_root'] ? "bisect job result directory:\n#{prefix_srv}#{ENV['result_root']}\n" : '' bisect_report_doc = "bisect email doc:\nhttps://gitee.com/wu_fengguang/compass-ci/blob/master/doc/bisect_emai…" - pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/#{@pkgbuild_repo}\n" + pkgbuild_repo_url = "PKGBUILD:\n#{prefix_srv}/git/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job result directory:\n#{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY -- 2.23.0

1 0

[PATCH lkp-tests] setup/simply-ci: fix failed to git clone
by Liu Yinsi 09 Feb '21

09 Feb '21

[error] Cloning into '/c/compass-ci'... fatal: unable to look up (port 9418) (Name or service not known) [how] before: git clone git://$GIT_SERVER/... after: git clone https://$GIT_SERVER/... Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index df18272f3..076c357d9 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone git://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } dev_env() -- 2.23.0

1 0

[PATCH compass-ci] docker/docker.rb: use curl instead of wget
by Lin Jiaxin 09 Feb '21

09 Feb '21

wget -q will turn off wget's output, even error. curl -sS can disable progress meter but still show error messages. So use curl -sS instead of wget -q. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- providers/docker/docker.rb | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 56c9754..f4e954e 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -49,8 +49,9 @@ def parse_response(url) return hash end -def wget_cmd(path, url, name) - system "wget -q -P #{path} #{url} && gzip -dc #{path}/#{name} | cpio -id -D #{path}" + +def curl_cmd(path, url, name) + system "curl -sS --create-dirs -o #{path}/#{name} #{url} && gzip -dc #{path}/#{name} | cpio -id -D #{path}" end def build_load_path(hostname) @@ -75,8 +76,8 @@ def load_initrds(load_path, hash) arch = RUBY_PLATFORM.split('-')[0] job_url = hash['job'] lkp_url = hash['lkp'] - wget_cmd(load_path, job_url, 'job.cgz') - wget_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") + curl_cmd(load_path, job_url, 'job.cgz') + curl_cmd(load_path, lkp_url, "lkp-#{arch}.cgz") end def start_container(hostname, load_path, hash) -- 2.23.0

2 1

[PATCH compass-ci] sbin/auto_submit: add support for different branch different job
by Li Yuanchao 08 Feb '21

08 Feb '21

Now can support some branch use specific submit commands and others use common submit commands for the same repo. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 78 ++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 62 insertions(+), 16 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 410445d..6d29543 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -30,21 +30,6 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) - newrefs_info['new_refs']['heads'].each do |branch, commit_id| - inactive_time = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git log --pretty=format:"%cr" -1 #{commit_id}) - next if inactive_time =~ /(month|year)/ - - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? - - system(real_argvs.join(' ')) - end - end - def get_argvs(newrefs_info) git_repo = newrefs_info['git_repo'] puts "git_repo : #{git_repo}" @@ -60,16 +45,77 @@ class AutoSubmit submit_argv end + def submit_one_job(submit_argv, git_repo, branch, commit_id) + inactive_time = %x(git -C /srv/git/#{git_repo}.git log --pretty=format:"%cr" -1 #{commit_id}) + return if inactive_time =~ /(month|year)/ + + real_argvs = Array.new(submit_argv) + real_argvs.push("upstream_branch=#{branch}") + real_argvs.push("upstream_commit=#{commit_id}") + tag = %x(git -C /srv/git/#{git_repo}.git tag --points-at #{commit_id}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + + system(real_argvs.join(' ')) + end + + def get_branch_commands(submit_commands) + submit_commands.each do |element| + return element['branch_commands'] if element['branch_commands'] + end + return nil + end + + def submit_specific_branch(submit_argv, newrefs_info, branch_commands) + return newrefs_info['new_refs']['heads'] if branch_commands.nil? + + non_specific_commits = {} + newrefs_info['new_refs']['heads'].each do |branch, commit_id| + branch_name = branch.delete_prefix('refs/heads/') + if branch_commands[branch_name].nil? + non_specific_commits[branch] = commit_id + next + end + argvs = Array.new(submit_argv) + argvs.push(branch_commands[branch_name]) + submit_one_job(argvs, newrefs_info['git_repo'], branch_name, commit_id) + end + return non_specific_commits + end + + def submit_non_specific(non_specific_commits, git_repo, submit_argv) + non_specific_commits.each do |branch, commit_id| + branch = branch.delete_prefix('refs/heads/') + submit_one_job(submit_argv, git_repo, branch, commit_id) + end + end + + # Add support of different branch different job + # newrefs_info['submit'] is like: + # ["submit command 1", + # "submit command 2", + # ..., + # { "branch_commands" => { "branch_1" => "submit command 1", + # ..., + # "branch_n" => "submit command n" + # } + # } + # ] def submit_job(newrefs_info) submit_argv = get_argvs(newrefs_info) return unless submit_argv return unless newrefs_info['submit'] + branch_commands = get_branch_commands(newrefs_info['submit']) + non_specific_commits = submit_specific_branch(submit_argv, newrefs_info, branch_commands) + return if non_specific_commits.empty? + newrefs_info['submit'].each do |argv_config| + next if argv_config.is_a? Hash + argvs = Array.new(submit_argv) argvs.push(argv_config) - submit(newrefs_info, argvs) + submit_non_specific(non_specific_commits, newrefs_info['git_repo'], argvs) end end -- 2.23.0

1 0

[PATCH v4 compass-ci 1/3] container/defconfig: add function docker_skip_rebuild
by Liu Yinsi 08 Feb '21

08 Feb '21

When execute `sparrow/4-docker/buildall run-only`, if docker image already exists, just exit build script, not rebuild container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/defconfig.rb | 6 ++++++ container/defconfig.sh | 8 ++++++++ 2 files changed, 14 insertions(+) diff --git a/container/defconfig.rb b/container/defconfig.rb index f9b0824..7dc9f33 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -47,3 +47,9 @@ def get_available_memory # take the middle value according to the system memory size. [1024, 30720, Math.sqrt(memtotal) * 1024].sort[1].to_i end + +def docker_skip_rebuild(tag) + return unless ENV['action'].include? 'run-only' + + exit 1 if system "docker image inspect #{tag} > /dev/null 2>&1" +end diff --git a/container/defconfig.sh b/container/defconfig.sh index 54e65b5..7c575ef 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -41,3 +41,11 @@ push_image() docker tag "$src_tag" "$dst_tag" docker push "$dst_tag" } + +docker_skip_rebuild() +{ + tag=$1 + [ "$action" != "run-only" ] && return + docker image inspect $tag > /dev/null 2>&1 + [ "$?" == "0" ] && exit 1 +} -- 2.23.0

1 0

[PATCH v4 compass-ci] container/register-accounts: fix failed to register account
by Liu Yinsi 08 Feb '21

08 Feb '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/register-accounts# ./start register-accounts register-accounts /c/compass-ci/container/register-accounts # ./register-accounts awk: /etc/compass-ci/defaults/*.yaml: No such file or directory "option: {\"my_email\"=>\"\", \"my_name\"=>\"\", \"lab\"=>\"\"}" Traceback (most recent call last): 10: from ./register-accounts:8:in `<main>' 9: from ./register-accounts:8:in `require_relative' 8: from /c/compass-ci/sbin/build-my-info.rb:57:in `<top (required)>' 7: from /c/compass-ci/sbin/build-my-info.rb:48:in `build_my_info' 6: from /c/compass-ci/lib/build_my_info_client.rb:30:in `search_my_info' 5: from /c/compass-ci/lib/es_query.rb:58:in `query_by_id' 4: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-api-7.10.1/lib/elasticsearch/api/actions/search.rb:103:in `search' 3: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/client.rb:182:in `perform_request' 2: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/http/faraday.rb:37:in `perform_request' 1: from /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:347:in `perform_request' /usr/lib/ruby/gems/2.7.0/gems/elasticsearch-transport-7.10.1/lib/elasticsearch/transport/transport/base.rb:218:in `__raise_transport_error': [400] {"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"accounts","node":"I1usO2axQ9-XZhzcX3oHxw","reason":{"type":"query_shard_exception","reason":"failed to create query: {\\n \\"bool\\" : {\\n \\"must\\" : [\\n {\\n \\"term\\" : {\\n \\"_id\\" : {\\n \\"value\\" : \\"\\",\\n \\"boost\\" : 1.0\\n }\\n }\\n }\\n ],\\n \\"adjust_pure_negative\\" : true,\\n \\"boost\\" : 1.0\\n }\\n}","index_uuid":"MH_AlDBvTYa9sQhwsESXgQ","index":"accounts","caused_by":{"type":"illegal_argument_exception","reason":"Ids can't be empty"}}}]},"status":400} (Elasticsearch::Transport::Transport::Errors::BadRequest) refer code: /c/compass-ci/sbin/build-my-info.rb: ... option = { 'my_email' => `git config --global user.email`.chomp, 'my_name' => `git config --global user.name`.chomp, 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp } ... build_my_info(option) ... [why] when call build-my-info.rb, we just want call function build_my_info(), no need to execute option, in caller script already define option values [how] use `if $PROGRAM_NAME == __FILE__`, when sbin/build-my-info.rb is not executed independently but invoked by other files, avoid to execute some part code. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/build-my-info.rb | 72 ++++++++++++++++++++++--------------------- 1 file changed, 37 insertions(+), 35 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index dbe093d..c7a7798 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -7,49 +7,51 @@ require_relative '../lib/es_client' require_relative '../lib/build_my_info_client' require 'optparse' -option = { - 'my_email' => `git config --global user.email`.chomp, - 'my_name' => `git config --global user.name`.chomp, - 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp -} +def build_my_info(option) + build_my_info = BuildMyInfo.new(option['my_email']) + info_es = build_my_info.search_my_info + info_es.update option + info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? -options = OptionParser.new do |opts| - opts.banner = "Usage: build-my-info [-e email] [-n name] [-l lab] [-t]\n" + build_my_info.config_my_info(info_es) +end - opts.separator '' - opts.separator 'options:' +if $PROGRAM_NAME == __FILE__ + option = { + 'my_email' => `git config --global user.email`.chomp, + 'my_name' => `git config --global user.name`.chomp, + 'lab' => `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp + } - opts.on('-e email', 'my_email') do |email| - option['my_email'] = email - end + options = OptionParser.new do |opts| + opts.banner = "Usage: build-my-info [-e email] [-n name] [-l lab] [-t]\n" - opts.on('-n name', 'my_name') do |name| - option['my_name'] = name - end + opts.separator '' + opts.separator 'options:' - opts.on('-l lab', 'lab') do |lab| - option['lab'] = lab - end + opts.on('-e email', 'my_email') do |email| + option['my_email'] = email + end - opts.on('-t', 'my_token') do - option['my_token'] = %x(uuidgen).chomp - end + opts.on('-n name', 'my_name') do |name| + option['my_name'] = name + end - opts.on_tail('-h', 'show this message') do - puts opts - exit - end -end + opts.on('-l lab', 'lab') do |lab| + option['lab'] = lab + end -options.parse! + opts.on('-t', 'my_token') do + option['my_token'] = %x(uuidgen).chomp + end -def build_my_info(option) - build_my_info = BuildMyInfo.new(option['my_email']) - info_es = build_my_info.search_my_info - info_es.update option - info_es['my_token'] = %x(uuidgen).chomp if info_es['my_token'].nil? + opts.on_tail('-h', 'show this message') do + puts opts + exit + end + end - build_my_info.config_my_info(info_es) -end + options.parse! -build_my_info(option) + build_my_info(option) +end -- 2.23.0

1 0

[PATCH lkp-tests] sbin/install-dependencies.sh: delete redundant code
by Wang Yong 08 Feb '21

08 Feb '21

write_shell_profile function will source profile, no need source twice Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 1 - 1 file changed, 1 deletion(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index ed8aeb927..b687dc0d6 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -65,7 +65,6 @@ set_env() write_host write_shell_profile "export LKP_SRC=$PWD" write_shell_profile "export PATH=\$PATH:\$LKP_SRC/sbin:\$LKP_SRC/bin" - source $(shell_profile) } set_env -- 2.23.0

1 0

[PATCH v3 lkp-tests] sbin/install-dependencies.sh: change install way
by Wang Yong 08 Feb '21

08 Feb '21

[before] install packages from $DISTRO detect [after] install packages from Package Manager it's not convenient when install packages in a new $DISTRO Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- sbin/install-dependencies.sh | 32 ++++++++++++++++++++------------ 1 file changed, 20 insertions(+), 12 deletions(-) diff --git a/sbin/install-dependencies.sh b/sbin/install-dependencies.sh index b5b070f01..ed8aeb927 100755 --- a/sbin/install-dependencies.sh +++ b/sbin/install-dependencies.sh @@ -5,28 +5,36 @@ PROJECT_DIR=$(dirname $SCRIPT_DIR) . $PROJECT_DIR/lib/env.sh -# choose install function base on DISTRIBUTION +# choose install function base on common Package Manager linux_dep() { - source /etc/os-release - case $ID in - ubuntu|debian) + get_package_manager + + case "$installer" in + apt-get) export DEBIAN_FRONTEND=noninteractive - sudo apt-get install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + sudo "$installer" install -yqm ruby-dev libssl-dev gcc g++ uuid-runtime + ;; + dnf|yum) + sudo "$installer" install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux ;; - openEuler|fedora|rhel|centos) - if type dnf > /dev/null 2>&1; then - sudo dnf install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - else - sudo yum install -y --skip-broken ruby rubygems gcc gcc-c++ make ruby-devel git lftp util-linux - fi + pacman) + sudo "$installer" -Sy --noconfirm --needed ruby rubygems gcc make git lftp util-linux ;; *) - echo "$ID not support! please install dependencies manually." && exit 1 + echo "Unknown Package Manager! please install dependencies manually." && exit 1 ;; esac } +get_package_manager() +{ + has_cmd "yum" && installer="yum" + has_cmd "dnf" && installer="dnf" && return + has_cmd "apt-get" && installer="apt-get" && return + has_cmd "pacman" && installer="pacman" && return +} + mac_dep() { brew install ruby -- 2.23.0

2 2