January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] lib/job.cr: add log when check account failed
by Wu Zhende 21 Jan '21

21 Jan '21

[Why] The information about the job that fails the account verification can be traced on kibana. [Log] {"level_num":2,"level":"WARN","time":"2021-01-21T11:28:25.476+0800", "msg":"Invalid account","my_email":"wuzhende666(a)163.com","my_name":"wuzhende","suite":"iperf","testbox":vm-2p8g"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 27 +++++++++++++++++++++------ 1 file changed, 21 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 17a0f1b..9b52d88 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -10,6 +10,7 @@ require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" require "../scheduler/elasticsearch_client" +require "./json_logger" struct JSON::Any def []=(key : String, value : String) @@ -53,6 +54,7 @@ class Job @hash = job_content.as_h @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new + @log = JSONLogger.new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" @@ -394,17 +396,30 @@ class Job error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) - raise error_msg unless account_info.is_a?(JSON::Any) + + flag = is_valid_account?(account_info) + @log.warn({"msg" => "Invalid account", + "my_email" => self["my_email"], + "my_name" => self["my_name"], + "suite" => self["suite"], + "testbox" => self["testbox"] + }.to_json) unless flag + raise error_msg unless flag + + @hash.delete("my_uuid") + @hash.delete("my_token") + end + + private def is_valid_account?(account_info) + return false unless account_info.is_a?(JSON::Any) @account_info = account_info.as_h # my_name can be nil in es # my_token can't be nil in es - raise error_msg unless self["my_name"] == @account_info["my_name"]?.to_s - raise error_msg unless self["my_token"] == @account_info["my_token"]? - - @hash.delete("my_uuid") - @hash.delete("my_token") + return false unless self["my_name"] == @account_info["my_name"]?.to_s + return false unless self["my_token"] == @account_info["my_token"]? + return true end private def get_initialized_keys -- 2.23.0

1 0

[PATCH v2 lkp-tests] sbin/submit: add -f option for uploading libvirt config
by Ren Wen 21 Jan '21

21 Jan '21

An option for 'submit': -f, --upload-file FILE upload an XML template of libvirt FILE's extension should be '.xml'. Example: case1: submit -f existent_file.xml => existent_file/<md5sum>.xml case2: submit -f nonexistent_file.xml => Find no file: nonexistent_file.xml Signed-off-by: Ren Wen <15991987063(a)163.com> --- lib/job.rb | 11 +++++++++++ sbin/submit | 31 +++++++++++++++++++++++++++++++ 2 files changed, 42 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 631ebfc98..812f8c299 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -16,6 +16,17 @@ require 'set' require 'pp' require 'English' +# copied from compass-ci/container/defconfig.rb +def cci_defaults + hash = {} + Dir.glob(['/etc/compass-ci/defaults/*.yaml', + '/etc/compass-ci/accounts/*.yaml', + "#{ENV['HOME']}/.config/compass-ci/defaults/*.yaml"]).each do |file| + hash.update YAML.load_file(file) || {} + end + hash +end + def restore(ah, copy) if ah.class == Hash ah.clear.merge!(copy) diff --git a/sbin/submit b/sbin/submit index 39cdcf576..e9e10d903 100755 --- a/sbin/submit +++ b/sbin/submit @@ -17,6 +17,7 @@ opt_auto_define_files = false opt_monitor = false opt_monitor_query = {} opt_my_queue = false +opt_upload_files = [] actions = ['output', 'stop'] options = OptionParser.new do |opts| @@ -64,10 +65,40 @@ options = OptionParser.new do |opts| opt_my_queue = true end + opts.on('-f FILE', '--upload-file FILE', "upload an XML template of libvirt. File: *.xml") do |file| + unless File.exist? file + puts "Find no file: #{file}" + exit 1 + end + + opt_upload_files << file + end + end options.parse!(ARGV) +def upload_configs(files) + defaults = cci_defaults() + upload_host = defaults['LKP_SERVER'] + upload_port = defaults['UPLOAD_LIBVIRT_PORT'] || 3081 + + files.each do |file| + response = `curl -sSf -T #{file} http://#{upload_host}:#{upload_port}/` + if $? == 0 + puts response + else + # TODO: how to deal with uploaded file when one failing + puts response + exit 1 + end + end + + exit 0 +end + +upload_configs(opt_upload_files) + seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' -- 2.23.0

1 1

[PATCH compass-ci] lib/job.cr: add log when check account failed
by Wu Zhende 21 Jan '21

21 Jan '21

[Why] The information about the job that fails the account verification can be traced on kibana. [Log] {"level_num":2,"level":"WARN","time":"2021-01-21T11:08:19.647+0800","msg":"Invalid account","my_email":"wuzhende666(a)163.com","suite":"iperf","testcase":"iperf"} Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 26 ++++++++++++++++++++------ 1 file changed, 20 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 17a0f1b..2971a77 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -10,6 +10,7 @@ require "scheduler/jobfile_operate.cr" require "scheduler/kernel_params.cr" require "scheduler/pp_params.cr" require "../scheduler/elasticsearch_client" +require "./json_logger" struct JSON::Any def []=(key : String, value : String) @@ -53,6 +54,7 @@ class Job @hash = job_content.as_h @es = Elasticsearch::Client.new @account_info = Hash(String, JSON::Any).new + @log = JSONLogger.new # init job with "-1", or use the original job_content["id"] id = "-1" if "#{id}" == "" @@ -394,17 +396,29 @@ class Job error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" account_info = @es.get_account(self["my_email"]) - raise error_msg unless account_info.is_a?(JSON::Any) + + flag = is_valid_account?(account_info) + @log.warn({"msg" => "Invalid account", + "my_email" => self["my_email"], + "suite" => self["suite"], + "testcase" => self["testcase"] + }.to_json) unless flag + raise error_msg unless flag + + @hash.delete("my_uuid") + @hash.delete("my_token") + end + + private def is_valid_account?(account_info) + return false unless account_info.is_a?(JSON::Any) @account_info = account_info.as_h # my_name can be nil in es # my_token can't be nil in es - raise error_msg unless self["my_name"] == @account_info["my_name"]?.to_s - raise error_msg unless self["my_token"] == @account_info["my_token"]? - - @hash.delete("my_uuid") - @hash.delete("my_token") + return false unless self["my_name"] == @account_info["my_name"]?.to_s + return false unless self["my_token"] == @account_info["my_token"]? + return true end private def get_initialized_keys -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: add new_refs_count by year/month/day
by Li Yuanchao 21 Jan '21

21 Jan '21

Add a hash table new_refs_count in fork_stat for every repo, which will record new refs times of this repo by year/month/day. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 43 +++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 41 insertions(+), 2 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 4f74751..6d1624f 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -298,6 +298,7 @@ class MirrorMain offset_new_refs = 0 if offset_new_refs >= 10 @fork_stat[git_repo][:new_refs_time][offset_new_refs] = Time.now.to_s @fork_stat[git_repo][:offset_new_refs] = offset_new_refs + 1 + @fork_stat[git_repo][:new_refs_count] = update_new_refs_count(@fork_stat[git_repo][:new_refs_count]) end def update_fork_stat(git_repo, possible_new_refs) @@ -373,15 +374,53 @@ class MirrorMain fetch_time: [], offset_fetch: 0, new_refs_time: [], - offset_new_refs: 0 + offset_new_refs: 0, + new_refs_count: {} } query = { query: { match: { git_repo: git_repo } } } result = @es_client.search(index: 'repo', body: query)['hits'] return fork_stat unless result['total'].positive? fork_stat.each_key do |key| - fork_stat[key] = result['hits'][0]['_source'][key.to_s] + fork_stat[key] = result['hits'][0]['_source'][key.to_s] || fork_stat[key] end return fork_stat end + + def create_year_hash(new_refs_count, year, month, day) + new_refs_count[year] = { 'year_count' => 1, month => { 'month_count' => 1, day => 1 } } + return new_refs_count + end + + def update_year_hash(new_refs_count, year, month, day) + new_refs_count[year]['year_count'] += 1 + return create_month_hash(new_refs_count, year, month, day) if new_refs_count[year][month].nil? + + return update_month_hash(new_refs_count, year, month, day) + end + + def create_month_hash(new_refs_count, year, month, day) + new_refs_count[year][month] = { 'month_count' => 1, day => 1 } + return new_refs_count + end + + def update_month_hash(new_refs_count, year, month, day) + new_refs_count[year][month]['month_count'] += 1 + if new_refs_count[year][month][day].nil? + new_refs_count[year][month][day] = 1 + else + new_refs_count[year][month][day] += 1 + end + return new_refs_count + end + + def update_new_refs_count(new_refs_count) + t = Time.now + year = t.year.to_s + month = t.month.to_s + day = t.day.to_s + return create_year_hash(new_refs_count, year, month, day) if new_refs_count[year].nil? + + return update_year_hash(new_refs_count, year, month, day) + end end -- 2.23.0

1 0

[PATCH compass-ci] container/assist-result: add check_job_credible function
by Cao Xueliang 21 Jan '21

21 Jan '21

The container assist-result mount "/srv/result" dir, the function credible? need "/srv/result" dir, so it can be implemented in the container. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 2 +- container/assist-result/routes.rb | 12 ++++++++++++ .../assist-result/views/check_job_credible.rb | 9 +++++++++ lib/assist_result_client.rb | 15 +++++++++++++++ 4 files changed, 37 insertions(+), 1 deletion(-) create mode 100644 container/assist-result/views/check_job_credible.rb diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile index 47ef92f..60a2fbf 100644 --- a/container/assist-result/Dockerfile +++ b/container/assist-result/Dockerfile @@ -12,4 +12,4 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou RUN umask 002 && \ echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport json yaml elasticsearch sinatra puma + gem install rest-client activesupport json yaml elasticsearch sinatra puma terminal-table diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 1013643..077a427 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,6 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' +require_relative './views/check_job_credible' configure do set :bind, '0.0.0.0' @@ -22,3 +23,14 @@ get '/get_job_yaml/:job_id' do return [200, result.to_json] end + +post '/check_job_credible' do + begin + data = JSON.parse(Base64.decode64(request.body.read)) + result = check_job_credible(data['pre_job_id'], data['cur_job_id'], data['error_id']) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, {'credible' => result}.to_json] +end diff --git a/container/assist-result/views/check_job_credible.rb b/container/assist-result/views/check_job_credible.rb new file mode 100644 index 0000000..a8d387f --- /dev/null +++ b/container/assist-result/views/check_job_credible.rb @@ -0,0 +1,9 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require_relative "#{ENV['CCI_SRC']}/lib/compare_error_messages" + +def check_job_credible(pre_job_id, cur_job_id, error_id) + return credible?(pre_job_id, cur_job_id, error_id) +end diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb index 042b189..fcf5d9d 100644 --- a/lib/assist_result_client.rb +++ b/lib/assist_result_client.rb @@ -3,6 +3,8 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true +require 'json' +require 'base64' require 'rest-client' require_relative 'constants' @@ -19,4 +21,17 @@ class AssistResult return response.body end + + def check_job_credible(pre_job_id, cur_job_id, error_id) + data = { + 'pre_job_id' => pre_job_id, + 'cur_job_id' => cur_job_id, + 'error_id' => error_id + } + resource = RestClient::Resource.new("http://#{@host}:#{@port}/check_job_credible") + response = resource.post(Base64.encode64(data.to_json)) + return nil unless response.code == 200 + + return JSON.parse(response.body) + end end -- 2.23.0

1 0

[PATCH compass-ci] /container/dracut-initrd/bin: refactor the code
by Xu Xijian 21 Jan '21

21 Jan '21

1. delete the redundant space before "-d" of "[ -d xx ]" 2. add blank line after "continue" cmd 3. turn some 2 line code block into one line to be more compact Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/overlay-lkp.sh | 11 +++++------ 1 file changed, 5 insertions(+), 6 deletions(-) diff --git a/container/dracut-initrd/bin/overlay-lkp.sh b/container/dracut-initrd/bin/overlay-lkp.sh index 1f5819b..9a2800e 100644 --- a/container/dracut-initrd/bin/overlay-lkp.sh +++ b/container/dracut-initrd/bin/overlay-lkp.sh @@ -2,7 +2,7 @@ # SPDX-License-Identifier: GPL-2.0 # transfer LKP dirs -[ -d /lkp ] || return 0 +[ -d /lkp ] || return 0 cp -a /lkp "$NEWROOT"/ @@ -11,8 +11,8 @@ do dir=$(basename "$i") [ "$i" != "${i%/\*}" ] && continue # skip: i='dir/*' - [ -d "$NEWROOT/$dir" ] || - mkdir -p "$NEWROOT/$dir" + + [ -d "$NEWROOT/$dir" ] || mkdir -p "$NEWROOT/$dir" for j in "$i"/* "$i"/.??* do @@ -25,8 +25,7 @@ do subdir=$(basename "$j") - [ -d "$NEWROOT/$dir/$subdir" ] || - mkdir -p "$NEWROOT/$dir/$subdir" + [ -d "$NEWROOT/$dir/$subdir" ] || mkdir -p "$NEWROOT/$dir/$subdir" for k in "$j"/* do @@ -37,7 +36,7 @@ do done done -[ -d /usr/src ] && +[ -d /usr/src ] && cp -a /usr/src "$NEWROOT"/usr/ kmdir=/lib/modules/$(uname -r) -- 2.23.0

2 1

[PATCH compass-ci] sparrow: use nokogiri to parse xml file
by Xiao Shenwei 21 Jan '21

21 Jan '21

[why] nokogiri dependent zlib-devel Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- sparrow/0-package/common | 2 +- sparrow/0-package/os/openEuler | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/sparrow/0-package/common b/sparrow/0-package/common index 2d7c9e3..1eb175c 100755 --- a/sparrow/0-package/common +++ b/sparrow/0-package/common @@ -2,7 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch +gem install --no-user-install git activesupport rest-client cucumber json faye-websocket elasticsearch nokogiri grep -q '^lkp:' /etc/passwd || useradd -u 1090 lkp grep -q '^mailer:' /etc/passwd || useradd -u 1144 mailer diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 61a37c4..e815fd1 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -24,6 +24,7 @@ pkgs=( libvirt curl sudo + zlib-devel ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..7f3d982 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,16 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 0

[PATCH compass-ci] upload-libvirt/nginx.conf: nginx.conf in upload-libvirt container
by cuiyili 21 Jan '21

21 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/upload-libvirt/nginx.conf | 43 +++++++++++++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 container/upload-libvirt/nginx.conf diff --git a/container/upload-libvirt/nginx.conf b/container/upload-libvirt/nginx.conf new file mode 100644 index 0000000..6dea3d3 --- /dev/null +++ b/container/upload-libvirt/nginx.conf @@ -0,0 +1,43 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user lkp; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + server { + listen 3081; + server_name "upload-libvirt"; + server_tokens off; + client_max_body_size 50m; + access_log /tmp/access.log; + error_log /tmp/error.log; + + location / { + allow all; + root /srv/cci/libvirt-xml/; + + autoindex on; + create_full_put_path on; + dav_methods PUT MKCOL; + dav_access user:rw group:rw all:rw; + content_by_lua_block { + ngx.req.read_body() + local filename = string.gsub(ngx.var.request_uri, "/", "") + if string.match(filename, "([^/]+).xml$") then + add_path = string.match(filename, "([^/]+).xml$") + local data = ngx.req.get_body_data() + local file = io.open("/srv/cci/libvirt-xml/"..filename, "w") + file:write(data) + file:close() + file_md5 = io.popen("cd /srv/cci/libvirt-xml/ && file_md5=$(md5sum "..filename.." | awk '{print $1}') && mkdir -p "..add_path.." && mv "..filename.." "..add_path.."/${file_md5}.xml && echo ${file_md5}"):read() + ngx.say(add_path.."/"..tostring(file_md5)..".xml") + end + } + } + + } +} -- 2.23.0

2 1

[PATCH v4 compass-ci] create-image: disable selinux
by Wang Chenglong 21 Jan '21

21 Jan '21

[Error] [ 5.786497] Run /sbin/init as init process SELinux: Could not open policy file <= /etc/selinux/targeted/policy/policy.32: No such file or directory Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 631bb14..411c130 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -13,6 +13,16 @@ setup_rootfs() ln -fs /usr/share/zoneinfo/Asia/Shanghai /etc/localtime rm -f /etc/machine-id /var/lib/dbus/machine-id systemd-machine-id-setup + + echo "Disable selinux" + if [ -f /etc/selinux/config ]; then + sed -i -r 's/^.*SELINUX=enforcing.*$/SELINUX=disabled/g' "/etc/selinux/config" + else + cat > "/etc/selinux/config" <<-EOF + SELINUX=disabled + SELINUXTYPE=targeted + EOF + fi } setup_login() { -- 2.23.0

1 0