- Compass-ci - mailweb.openeuler.org

[PATCH v5 lkp-tests] stats/ansible_test: stats result for ansible migrating
by Wang Chenglong 25 Feb '21

25 Feb '21

[How] ansible_stats_openeuler < ${job_output} wcl@crystal ~/lkp-tests/stats% ./ansible_test < /srv/result/ansible_test/2021-02-04/ vm-2p16g--wcl/openeuler-20.03-aarch64/jgeusebroek-ansible-role-docker/crystal.974042/ansible_test error.Failed-to-download-metadata-for-repo-dockerrepo: 1 error.Failed-to-download-metadata-for-repo-dockerrepo.message: {"ansible_loop_var": "item", "changed": false, "item": "docker-engine", "msg": "Failed to download metadata for repo 'dockerrepo': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried", "rc": 1, "results": []} Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- stats/ansible_test | 138 +++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 127 insertions(+), 11 deletions(-) diff --git a/stats/ansible_test b/stats/ansible_test index 4c7d07b3e..67e312f47 100755 --- a/stats/ansible_test +++ b/stats/ansible_test @@ -1,15 +1,131 @@ #!/usr/bin/env ruby -while (line = STDIN.gets) - case line.chomp! - when /ok=(\d+)\s+changed=(\d+)\s+unreachable=(\d+)\s+failed=(\d+)\s+skipped=(\d+)\s+rescued=(\d+)\s+ignored=(\d+)/ - puts "ansible_test.total.nr_ok: #{$1}" - puts "ansible_test.total.nr_changed: #{$2}" - puts "ansible_test.total.nr_failed: #{$4}" - puts "ansible_test.total.nr_skipped: #{$5}" - puts "ansible_test.total.nr_rescued: #{$6}" - puts "ansible_test.total.nr_ignored: #{$7}" - when /playbook_run_on_fail/ - puts line +require 'json' + +def output_error(error_msg) + error_id = common_error_id error_msg + puts "error.#{error_id}: 1" + puts "error.#{error_id}.message: #{@ansible_failed_info}" +end + +def parse_msg_1(ansible_failed_json) + case ansible_failed_json.chomp + + # When running some plug-in, the file was not found + when /(No file was found when using (.+))\./ + output_error $1 + + # {"changed": false, "cmd": "csf -v", "msg": "[Errno 2] No such file or directory: b'csf': b'csf'", "rc": 2} + when /(No such file or directory: b'.+')/ + output_error $1 + + # The task includes an option with an undefined variable + # {"msg": "The task includes an option with an undefined variable. The error was: '__mysql_packages' is undefined + # The error appears to be in '/root/.ansible/roles/ansible-role-mysql/tasks/variables.yml': line 11, column 3, + # but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: - name: + # Define mysql_packages. here"} + when /(.+ is undefined)/ + output_error $1 + + # {"changed": true, "cmd": ["packaging/installer/install-required-packages.sh", "--non-interactive", "netdata"], + # "msg": "non-zero return code", "rc": 1} + when /(non-zero return code)/ + output_error $1 + + # Some service could not find the requested host + when /(Could not find the requested service .+): host/ + output_error $1 + + # Unable to start some services + when /(Unable to start service .+):/ + output_error $1 + end +end + +def parse_msg_2(ansible_failed_json) + case ansible_failed_json.chomp + + # {"changed": false, "msg": "Failed to download packages: Cannot download 7.10.2/filebeat-7.10.2-aarch64.rpm: + # All mirrors were tried", "results": []} + when /(Failed to download packages: Cannot download .+):/ + output_error $1 + + when /(Failure downloading .+) / + output_error $1 + + # Failed to download the yum repodata + # {"changed": false, "msg": "Failed to download metadata for repo 'docker-ce-stable': Cannot download repomd.xml", + # "rc": 1, "results": []} + when /(Failed to download metadata for repo.+)':/ + output_error $1 + + # Some role in not sipported this system + # {"changed": false, "msg": "Distribution openEuler is not supported by this role!"} + when /(.+ is not supported .+)/ + output_error $1 + + # failed to create temporary content file: The read operation timed out. + when /(.+ The read operation timed out)/ + output_error $1 + + # {"msg": "The conditional check 'java_version is defined' failed. The error was: error while evaluating conditional + # (java_version is defined): {{ java_default_version }}: {{ _java_default_version[ansible_distribution ~ '-' ~ + # ansible_distribution_major_version] | default(_java_default_version[ansible_distribution]) }}: 'dict object' has + # no attribute 'openEuler'\n\nThe error appears to be in '/root/.ansible/roles/ansible-role-java/tasks/assert.yml': + # line 19, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line + # appears to be:\n\n\n- name: test if java_version is set correctly\n ^ here\n"} + when /(The conditional check .+ failed)/ + output_error $1 + end +end + +def parse_failures(ansible_failed_json) + ansible_failed_json.each do |items| + case items.chomp + # Failed to install some of the specified packages. + # {"changed": false, "failures": ["No package erlang available."], "msg": + # "Failed to install some of the specified packages", "rc": 1, "results": []} + when /(No package .+ available)/ + output_error $1 + end + end +end + +def parse_message(ansible_failed_json) + case ansible_failed_json.chomp + + # Could not find or access ansible yaml + # {"ansible_facts": {}, "ansible_included_var_files": [], "changed": false, "message": + # "Could not find or access 'openEuler.yml' Searched in: + # /root/.ansible/roles/ansible-unzip/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/openEuler.yml i + # /root/.ansible/roles/ansible-unzip/tasks/vars/openEuler.yml /root/.ansible/roles/ansible-unzip/tasks/openEuler.yml + # /root/.ansible/vars/openEuler.yml /root/.ansible/openEuler.yml on the Ansible Controller. + # If you are using a module and expect the file to exist on the remote, see the remote_src option"} + when /(Could not find or access .+')/ + output_error $1 end end + +def common_error_id(line) + line = line.chomp + line.gsub!(/ /, '-') + line.gsub!(/:/, '') # error.:-'__mysql_packages'-is-undefined: 1 + line.gsub!(/,/, '') + line.gsub!(/'/, '') # Failed-to-download-metadata-for-repo-'dockerrepo: 1 + line.gsub!(/!/, '') # Distribution-openEuler-is-not-supported-by-this-role!: 1 + line +end + +while (line = STDIN.gets) + next unless line =~ /(FAILED!|failed:).*=>(.*)/ + + @ansible_failed_info = $2 + next if @ansible_failed_info.empty? + + ansible_failed_json = JSON.parse @ansible_failed_info + + parse_msg_1 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_msg_2 ansible_failed_json['msg'] unless ansible_failed_json['msg'].nil? + parse_failures ansible_failed_json['failures'] unless ansible_failed_json['failures'].nil? + parse_message ansible_failed_json['message'] unless ansible_failed_json['message'].nil? +end -- 2.23.0

2 4

[PATCH v4 lkp-tests] jobs: add os-multi-dimension-*.yaml
by Li Ping 24 Feb '21

24 Feb '21

1 0

[PATCH compass-ci] sparrow/4-docker/buildall: fix continue error
by Liu Yinsi 24 Feb '21

24 Feb '21

[error] ./buildall: line 59: continue: only meaningful in a `for', `while', or `until' loop [how] before: [ "$container_name" == 'ssh-r' ] && continue after: [ "$container_name" == 'ssh-r' ] && exit [why] 'continue' can not use independently without `for', `while', or `until', so use 'exit' to fix. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/4-docker/buildall | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 96c7385..bb77a1f 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -56,7 +56,7 @@ do_one_run() mkdir $tmpdir/start_$container_name 2>/dev/null && ( cd "$container" - [ "$container_name" == 'ssh-r' ] && continue + [ "$container_name" == 'ssh-r' ] && exit [ -x first-run ] && ./first-run [ -x start ] && ./start [ "$container_name" == 'initrd-lkp' ] && ./run -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/es/start: call check_es_ready()
by Liu Yinsi 24 Feb '21

24 Feb '21

call container/defconfig.sh:check_es_ready() to accelerate es/logging-es startup time. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/es/start | 7 +++---- container/logging-es/start | 5 ++--- 2 files changed, 5 insertions(+), 7 deletions(-) diff --git a/container/es/start b/container/es/start index d84ad33..a1b81f9 100755 --- a/container/es/start +++ b/container/es/start @@ -20,7 +20,6 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - -set_es_indices +# set index +check_es_ready 9200 +find $CCI_SRC/sbin/ -name "es-*-mapping.sh" -exec sh {} \; diff --git a/container/logging-es/start b/container/logging-es/start index 66464e6..13bf25d 100755 --- a/container/logging-es/start +++ b/container/logging-es/start @@ -20,9 +20,8 @@ cmd=( "${cmd[@]}" -echo "Please wait 60s for setting index" -sleep 60 - +# set index +check_es_ready 9202 curl -sSH "Content-Type: application/json" -XPUT "localhost:9202/_settings" -d ' { "index": { -- 2.23.0

1 0

[PATCH lkp-tests] setup/simply-ci: fix git clone error
by Liu Yinsi 24 Feb '21

24 Feb '21

[how] before: git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git after: git clone https://gitee.com/wu_fengguang/compass-ci.git [why] refer code: git_ci(): ... if [ -n "$GIT_SERVER" ]; then cat >> /etc/gitconfig <<-EOF [url "git://$GIT_SERVER/gitee.com"] insteadOf=https://gitee.com ... if $GIT_SERVER exists, "http://gitee.com" will be instead of "git://$GIT_SERVER/gitee.com", so the old code is wrong. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 076c357d9..c8f8a5a53 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -14,7 +14,7 @@ git_ci() EOF fi - git clone https://$GIT_SERVER/gitee.com/wu_fengguang/compass-ci.git /c/compass-ci + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci } dev_env() -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 18 ++++-------------- 2 files changed, 12 insertions(+), 14 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..dbc42ef 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -125,30 +125,20 @@ module Utils File.open(commit_jobs, 'a+') { |f| f.puts content } end - def get_account_info - ESQuery.new(index: 'accounts').query_by_id(DELIMITER_EMAIL) - end - def init_job_content(job_id) job_yaml = AssistResult.new.get_job_yaml(job_id) raise "get job yaml failed, job id: #{job_id}" unless job_yaml job = JSON.parse job_yaml - - account_info = get_account_info - raise "query delimiter account info failed: #{DELIMITER_EMAIL}" unless account_info - record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] sched: response ipxe_response value if ipxe_response field in job yaml
by Cao Xueliang 24 Feb '21

24 Feb '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/scheduler/find_job_boot.cr | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 9e2898e..e8d590b 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -160,6 +160,8 @@ class Sched end private def get_boot_ipxe(job : Job) + return job["ipxe_response"] if job["ipxe_response"]? + response = "#!ipxe\n\n" _initrds_uri = Array(String).from_json(job.initrds_uri).map { |uri| "initrd #{uri}" } -- 2.23.0

2 2

[PATCH v2 compass-ci 3/3] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 10 ++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..cd026f9 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -141,14 +141,12 @@ module Utils record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] lib/git_bisect.rb: fix: syntax error, unexpected string literal, expecting']'
by Cao Xueliang 24 Feb '21

24 Feb '21

need add ',' after 'upstream_url' => @upstream_url Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index dd0b782..6aedda5 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -113,7 +113,7 @@ class GitBisect 'work_dir' => @work_dir, 'bisect_error' => bisect_error, 'all_errors' => all_errors, - 'upstream_url' => @upstream_url + 'upstream_url' => @upstream_url, 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end -- 2.23.0

1 0

[PATCH compass-ci 3/3] delimiter: use secrets field to submit process job
by Cao Xueliang 24 Feb '21

24 Feb '21

Before: Search account info from ES. After: Use secrets field to get the my_token. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter.rb | 8 ++++++++ src/delimiter/utils.rb | 10 ++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter.rb b/src/delimiter.rb index 1ea5de5..cb43cb4 100644 --- a/src/delimiter.rb +++ b/src/delimiter.rb @@ -5,8 +5,16 @@ require_relative './delimiter/delimiter' require_relative '../lib/config_account' +def config_secrets_yaml + account = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/defaults/account.yaml" + lab = YAML.load_file "#{ENV['HOME']}/.config/compass-ci/include/lab/#{account['lab']}.yaml" + secrets = Hash["secrets" => lab] + File.open("#{ENV['HOME']}/.config/compass-ci/defaults/secrets.yaml", "w") { |f| YAML.dump(secrets, f) } +end + begin config_yaml('delimiter') + config_secrets_yaml delimiter = Delimiter.new delimiter.start_delimit rescue StandardError => e diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 0930dd7..cd026f9 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -141,14 +141,12 @@ module Utils record_jobs(job['id'], job['upstream_commit']) job['suite'] = 'bisect' - job['my_name'] = account_info['my_name'] - job['my_email'] = account_info['my_email'] - job['my_token'] = account_info['my_token'] + job['my_name'] = ENV['my_name'] + job['my_email'] = ENV['my_email'] + job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id + job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" - job.delete('tboxgroup') - job.delete('subqueue') - job.delete('queue') job.delete('id') return job -- 2.23.0

1 0