- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] fix: change tool's name to docker2rootfs
by Wang Chenglong 11 Jan '21

11 Jan '21

[Why] Change tool's name to docker2rootfs, like iso2rootfs. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/{docker-rootfs => docker2rootfs}/README.md | 0 container/{docker-rootfs => docker2rootfs}/common | 0 container/{docker-rootfs => docker2rootfs}/packages-to-install | 0 container/{docker-rootfs => docker2rootfs}/run | 0 container/{docker-rootfs => docker2rootfs}/setup-image | 0 5 files changed, 0 insertions(+), 0 deletions(-) rename container/{docker-rootfs => docker2rootfs}/README.md (100%) rename container/{docker-rootfs => docker2rootfs}/common (100%) rename container/{docker-rootfs => docker2rootfs}/packages-to-install (100%) rename container/{docker-rootfs => docker2rootfs}/run (100%) rename container/{docker-rootfs => docker2rootfs}/setup-image (100%) diff --git a/container/docker-rootfs/README.md b/container/docker2rootfs/README.md similarity index 100% rename from container/docker-rootfs/README.md rename to container/docker2rootfs/README.md diff --git a/container/docker-rootfs/common b/container/docker2rootfs/common similarity index 100% rename from container/docker-rootfs/common rename to container/docker2rootfs/common diff --git a/container/docker-rootfs/packages-to-install b/container/docker2rootfs/packages-to-install similarity index 100% rename from container/docker-rootfs/packages-to-install rename to container/docker2rootfs/packages-to-install diff --git a/container/docker-rootfs/run b/container/docker2rootfs/run similarity index 100% rename from container/docker-rootfs/run rename to container/docker2rootfs/run diff --git a/container/docker-rootfs/setup-image b/container/docker2rootfs/setup-image similarity index 100% rename from container/docker-rootfs/setup-image rename to container/docker2rootfs/setup-image -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: fix submit job failed
by Liu Yinsi 11 Jan '21

11 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user intall compass-ci at their local server, need to verify when submit job in auto-submit container, so store autosubmit_info to ES, and mount yaml file in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- sbin/build-my-info.rb | 4 ++++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index fd75f51..ea581ca 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -31,3 +31,7 @@ options.parse! build_my_info = BuildMyInfo.new(option[:my_email], option[:my_name], option[:lab]) build_my_info.config_my_info + +autosubmit_info = YAML.load_file('/etc/compass-ci/account/autosubmit.yaml') +build_autosubmit_info = BuildMyInfo.new(autosubmit_info['my_email'], autosubmit_info['my_name'], autosubmit_info['lab']) +build_autosubmit_info.store_account_info -- 2.23.0

1 0

[PATCH v7 compass-ci 2/2] lib/es_query.rb: parse es_result to a Hash result
by Lu Kaiyi 11 Jan '21

11 Jan '21

[why] make es result more easy access. [example] input: es_result = [{"key"=>"build-pkg", "doc_count"=>354526, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>5708}, {"key"=>"finished", "doc_count"=>1033}, {"key"=>"incomplete", "doc_count"=>204}, {"key"=>"submit", "doc_count"=>136}, ... output: result_hash = {"build-pkg"=> {"failed"=>5708, "finished"=>1033, "incomplete"=>204, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1785, "failed"=>675, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 46 +++++++++++++++++++++++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 91aa6ab..928bd68 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -128,7 +128,7 @@ class ESQuery result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] return nil if result.empty? - result + parse_fields(result) end end @@ -216,3 +216,47 @@ def build_aggs_from_fields(fields) aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs aggs_hash end + +# input: +# es_result = +# [{"key"=>"build-pkg", +# "doc_count"=>354526, +# "all_job_state"=> +# {"doc_count_error_upper_bound"=>0, +# "sum_other_doc_count"=>0, +# "buckets"=> +# [{"key"=>"failed", "doc_count"=>5708}, +# {"key"=>"finished", "doc_count"=>1033}, +# {"key"=>"incomplete", "doc_count"=>204}, +# {"key"=>"submit", "doc_count"=>136}, +# ... +# +# output: +# result_hash = +# {"build-pkg"=> +# {"failed"=>5708, +# "finished"=>1033, +# "incomplete"=>204, +# "submit"=>136, +# "OOM"=>11, +# "post_run"=>2}, +# "cci-depends"=> +# {"finished"=>1785, +# "failed"=>675, +# ... +def parse_fields(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + sub_field = result.keys.detect { |field| field.start_with?('all_') } + + if sub_field + all_field = result[sub_field]['buckets'] + result_hash[key] = parse_fields(all_field) + else + result_hash[key] = result['doc_count'] + end + end + + result_hash +end -- 2.23.0

1 0

[PATCH v7 compass-ci 1/2] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 11 Jan '21

11 Jan '21

[why] get multi-field aggregation count for overview of report page. [input]: fields, query_items (optional for query_items, default no scope limitation) [example] input: fields = ['suite'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186214}, {"key"=>"cci-depends", "doc_count"=>1230}, {"key"=>"iperf", "doc_count"=>1103}, {"key"=>"spinlock", "doc_count"=>1000}, {"key"=>"cci-makepkg", "doc_count"=>949}, {"key"=>"iperf-walk-os-test", "doc_count"=>474}, {"key"=>"bisect", "doc_count"=>381}, ... input: fields = ['suite', 'job_state'] query_items = { 'os' => 'openeuler' } output: [{"key"=>"build-pkg", "doc_count"=>186227, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3830}, {"key"=>"finished", "doc_count"=>803}, {"key"=>"incomplete", "doc_count"=>196}, {"key"=>"submit", "doc_count"=>136}, ... input: fields = ['os', 'os_version', 'job_state'] query_items = {} output: [{"key"=>"openeuler", "doc_count"=>193287, "all_os_version"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"20.03", "doc_count"=>191789, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>3909}, {"key"=>"finished", "doc_count"=>2401}, {"key"=>"incomplete", "doc_count"=>300}, {"key"=>"submit", "doc_count"=>136}, {"key"=>"OOM", "doc_count"=>23}, {"key"=>"post_run", "doc_count"=>1}]}}, {"key"=>"20.03-SP1", "doc_count"=>904, "all_job_state"=> {"doc_count_error_upper_bound"=>0, "sum_other_doc_count"=>0, "buckets"=> [{"key"=>"failed", "doc_count"=>641}, ... Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 70 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 70 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..91aa6ab 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,49 @@ class ESQuery result end + + # input: + # fields, query_items + # + # fields => fields: Array(keyword) + # eg: + # ['suite', 'job_state'] + # + # query_items => query_items: Hash(keyword, value) + # eg: + # { + # 'os' => 'openeuler', + # ... + # } + # (optional for query_items, default no scope limitation) + # output: + # [{"key"=>"build-pkg", + # "doc_count"=>186175, + # "all_job_state"=> + # {"doc_count_error_upper_bound"=>0, + # "sum_other_doc_count"=>0, + # "buckets"=> + # [{"key"=>"failed", "doc_count"=>3830}, + # {"key"=>"finished", "doc_count"=>803}, + # {"key"=>"incomplete", "doc_count"=>196}, + # ... + def query_fields(fields, query_items = {}) + field1 = fields.first + aggs_hash = build_aggs_from_fields(fields) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: aggs_hash['aggs'], + size: 0 + } + result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + return nil if result.empty? + + result + end end # Range Query Example: @@ -146,3 +189,30 @@ def assign_desc_body(keyword) }] } end + +# input: +# fields = ['os', 'os_version', 'job_state'] +# output: +# aggs_hash = { +# "aggs"=> +# {"all_os"=> +# {"terms"=>{:field=>"os", :size=>1000}, +# "aggs"=> +# {"all_os_version"=> +# {"terms"=>{:field=>"os_version", :size=>1000}, +# "aggs"=> +# {"all_job_state"=>{"terms"=>{:field=>"job_state", :size=>1000}}}}}}} +# } +def build_aggs_from_fields(fields) + aggs_hash = {} + return if fields.empty? + + field = fields.shift + aggs_hash['aggs'] ||= {} + aggs_hash['aggs']["all_#{field}"] = { + 'terms' => { field: field, size: 1000 } + } + sub_aggs = build_aggs_from_fields(fields) + aggs_hash['aggs']["all_#{field}"].merge!(sub_aggs) if sub_aggs + aggs_hash +end -- 2.23.0

1 0

Re: [PATCH lkp-tests] jobs: and ansible_test.yaml
by Bai Jing 11 Jan '21

11 Jan '21

^^ add Thanks, Baijing Reply-To: In-Reply-To: <20210109124854.6869-1-ylzhangah(a)qq.com> On Sat, Jan 09, 2021 at 08:48:54PM +0800, Zhang Yale wrote: >Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> >--- > jobs/ansible_test.yaml | 9 +++++++++ > 1 file changed, 9 insertions(+) > create mode 100644 jobs/ansible_test.yaml > >diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml >new file mode 100644 >index 00000000..9adc4d71 >--- /dev/null >+++ b/jobs/ansible_test.yaml >@@ -0,0 +1,9 @@ >+suite: ansible_test >+category: functional >+ >+ansible_test: >+ username: #geerlingguy >+ rolename: #jenkins >+ ansible_host: #localhost >+ ansible_user: #root >+ ansible_python_interpreter: #/user/bin/python3 >-- >2.23.0 >

2 1

[PATCH lkp-tests] tests: add ansible_tests script
by Zhang Yale 11 Jan '21

11 Jan '21

This script can be invoked to run the ansible command to deploy ansible-roles which you specified in your environment. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) create mode 100755 tests/ansible_test diff --git a/tests/ansible_test b/tests/ansible_test new file mode 100755 index 00000000..09e5ca9e --- /dev/null +++ b/tests/ansible_test @@ -0,0 +1,59 @@ +#!/bin/bash +# username +# rolename +# ansible_host +# ansible_user +# ansible_python_interpreter + +[ -n "$username" ] || echo "username is empty" +[ -n "$rolename" ] || echo "rolename is empty" +[ -n "$ansible_host" ] || $ansible_host=localhost +[ -n "$ansible_user" ] || $ansible_user=root +[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 + +build_mutual_trust() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa + cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys + ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' +} + +install_ansible() +{ + pip3 install ansible + ansible localhost -m ping + #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko + [ $? -eq 0 ] && echo "Ansible is successfully deployed!" +} + +download_ansible_role() +{ + ansible-galaxy install $username.$rolename + [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" +} + +ansible_playbook_run() +{ + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible + cat <<-EOF > "$ansible_path"/hosts + $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter + EOF + [ $? -eq 0 ] && echo "Hostfile has been generated!" + + cat <<-EOF > "$ansible_path"/$rolename.yml + - hosts: $ansible_host + remote_user: $ansible_user + roles: + - role: $username.$rolename + EOF + [ $? -eq 0 ] && echo "Playbook has been generated!" + + ansible-playbook "$ansible_path"/$rolename.yml + [ $? -eq 0 ] && echo "Your playbook is running successfully!" +} + +#build_mutual_trust +install_ansible +download_ansible_role +ansible_playbook_run -- 2.23.0

5 10

[PATCH lkp-tests] sbin/submit: modify job_id to job id
by Hu Xuejiao 11 Jan '21

11 Jan '21

[why] Because we use "es-find (id='')" to search id-information, but when submit, it is (job_id=''). so it is recommended that the value be the same as the ID Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/submit | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/submit b/sbin/submit index a92556ff..448d6243 100755 --- a/sbin/submit +++ b/sbin/submit @@ -164,10 +164,10 @@ ARGV.each do |jobfile| JSON.parse(messages).each do |msg| if msg['message'].empty? job_ids << msg['job_id'].to_s - puts("submit #{jobfile}, got job_id=#{msg['job_id']}") + puts("submit #{jobfile}, got job id=#{msg['job_id']}") else opt_monitor = false - puts("submit #{jobfile} failed, got job_id=#{msg['job_id']}, error: #{msg['message']}") + puts("submit #{jobfile} failed, got job id=#{msg['job_id']}, error: #{msg['message']}") end end end -- 2.23.0

2 1

[PATCH lkp-tests] jobs: and ansible_test.yaml
by Zhang Yale 10 Jan '21

10 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- jobs/ansible_test.yaml | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 jobs/ansible_test.yaml diff --git a/jobs/ansible_test.yaml b/jobs/ansible_test.yaml new file mode 100644 index 00000000..9adc4d71 --- /dev/null +++ b/jobs/ansible_test.yaml @@ -0,0 +1,9 @@ +suite: ansible_test +category: functional + +ansible_test: + username: #geerlingguy + rolename: #jenkins + ansible_host: #localhost + ansible_user: #root + ansible_python_interpreter: #/user/bin/python3 -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends: add depends for openeuler_docker test
by Bai Jing 09 Jan '21

09 Jan '21

Signed-off-by: Bai Jing <799286817(a)qq.com> --- distro/depends/openeuler_docker | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 distro/depends/openeuler_docker diff --git a/distro/depends/openeuler_docker b/distro/depends/openeuler_docker new file mode 100644 index 000000000..b31074ad9 --- /dev/null +++ b/distro/depends/openeuler_docker @@ -0,0 +1,3 @@ +docker.io +wget +git -- 2.23.0

1 0

[PATCH compass-ci] qemu/kvm.sh: modify the prompt information of kernel
by Hu Xuejiao 09 Jan '21

09 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- providers/qemu/kvm.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..577bca1 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -59,7 +59,7 @@ parse_ipxe_script() check_option_value() { [ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" + echo "The kernel does not exist" exit } -- 2.23.0

1 0

[PATCH compass-ci 3/3] build-my-info: fix name mismatches
by Wu Fengguang 09 Jan '21

09 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index d0ee46e..fd75f51 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -15,11 +15,11 @@ option = { options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| - option[:email] = email + option[:my_email] = email end opts.on('-n name', 'my_name') do |name| - option[:name] = name + option[:my_name] = name end opts.on('-l lab', 'lab') do |lab| -- 2.23.0

1 0

[PATCH compass-ci 2/3] build-my-info: set default values
by Wu Fengguang 09 Jan '21

09 Jan '21

Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index 0b16884..d0ee46e 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -7,7 +7,12 @@ require_relative '../lib/es_client' require_relative '../lib/build_my_info_client' require 'optparse' -option = {} +option = { + my_name: `git config --global user.name`.chomp, + my_email: `git config --global user.email`.chomp, + lab: `awk '/^lab:\s/ {print $2; exit}' /etc/compass-ci/defaults/*.yaml`.chomp +} + options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| option[:email] = email -- 2.23.0

1 0

[PATCH compass-ci 1/3] build-my-info: fix white space style
by Wu Fengguang 09 Jan '21

09 Jan '21

Layout/SpaceAroundOperators: Surrounding space missing for operator `=`. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- sbin/build-my-info.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/build-my-info.rb b/sbin/build-my-info.rb index cd824c6..0b16884 100755 --- a/sbin/build-my-info.rb +++ b/sbin/build-my-info.rb @@ -8,7 +8,7 @@ require_relative '../lib/build_my_info_client' require 'optparse' option = {} -options= OptionParser.new do |opts| +options = OptionParser.new do |opts| opts.on('-e email', 'my_email') do |email| option[:email] = email end -- 2.23.0

1 0

[PATCH v2 compass-ci] container/openresty-proxy-cache: add nginx.conf
by cuiyili 08 Jan '21

08 Jan '21

http/https cache proxy and request forwarding processing Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/nginx.conf | 118 +++++++++++++++++++++ 1 file changed, 118 insertions(+) create mode 100755 container/openresty-proxy-cache/nginx.conf diff --git a/container/openresty-proxy-cache/nginx.conf b/container/openresty-proxy-cache/nginx.conf new file mode 100755 index 0000000..2e83286 --- /dev/null +++ b/container/openresty-proxy-cache/nginx.conf @@ -0,0 +1,118 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user nobody; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + proxy_cache_path /nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=10g + inactive=6000m use_temp_path=off; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + '"$upstream_cache_status" $body_bytes_sent'; + access_log /tmp/access.log main; + error_log /tmp/error.log; + + resolver 8.8.8.8; + client_max_body_size 700m; + server_tokens off; + + proxy_cache my_cache; + proxy_ignore_headers Set-Cookie Cache-Control Expires; + proxy_cache_valid 200 1440m; + proxy_cache_valid 301 302 5m; + proxy_set_header X-Real-Ip $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + add_header X-Cache $upstream_cache_status; + add_header X-Via $server_addr; + + server { + listen 443 ssl; + ssl_certificate /cert/ca.crt; + ssl_certificate_key /cert/ca.key; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + proxy_ssl_server_name on; + set_by_lua_block $cur_ups { + uri = ngx.var.request_uri + host = ngx.var.http_host + + path_file = "/srv/initrd/tar/"..uri + local f = io.open(path_file) + if f then + return "/127.0.0.1:443/tar/"..uri + end + return uri + } + proxy_pass $scheme:/$cur_ups; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } + server { + listen 80; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + + content_by_lua_block { + local http = require "resty.http" + local httpc = http.new() + uri = ngx.var.request_uri + host = ngx.var.http_host + if string.find(host, "11333") then + h="" + else + h="/"..host + end + path_file = "/srv/initrd/tar/"..h..uri + + local f = io.open(path_file) + if f then + url_forward = "http://127.0.0.1:80/tar/"..h..uri + else + url_forward = "http:/"..h..uri + end + + res, error = httpc:request_uri(url_forward) + if res then + ngx.status = res.status + for k, v in pairs(res.headers) do + if k ~= "Transfer-Encoding" and k ~= "Connection" then + ngx.header[k] = v + end + end + ngx.say(res.body) + else + ngx.exit(ngx.HTTP_FORBIDDEN) + end + httpc:close() + + } + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } +} -- 2.23.0

1 0

[PATCH v2 compass-ci] container/openresty-proxy-cache: add start script
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/start | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 container/openresty-proxy-cache/start diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start new file mode 100755 index 0000000..a1f095d --- /dev/null +++ b/container/openresty-proxy-cache/start @@ -0,0 +1,22 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm openresty-proxy-cache + +cmd=( + docker run -d + --name openresty-proxy-cache + --restart=always + -p 11333:80 + -p 11334:443 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /srv/initrd/tar:/srv/initrd/tar:ro + -v /etc/localtime:/etc/localtime:ro + openresty:proxy_cache +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH compass-ci] lua_script/forward.lua: http/https request forwarding processing
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- .../lua_script/forward.lua | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100755 container/openresty-proxy-cache/lua_script/forward.lua diff --git a/container/openresty-proxy-cache/lua_script/forward.lua b/container/openresty-proxy-cache/lua_script/forward.lua new file mode 100755 index 0000000..bd54742 --- /dev/null +++ b/container/openresty-proxy-cache/lua_script/forward.lua @@ -0,0 +1,32 @@ +local http = require "resty.http" +local httpc = http.new() +uri = ngx.var.request_uri +host = ngx.var.http_host +if string.find(host, "11333") then + h="" +else + h="/"..host +end +path_file = "/srv/initrd/tar/"..h..uri + +local f = io.open(path_file) +if f then + url_forward = "http://127.0.0.1:80/tar/"..h..uri +else + url_forward = "http:/"..h..uri +end + +res, error = httpc:request_uri(url_forward) +if res then + ngx.status = res.status + for k, v in pairs(res.headers) do + if k ~= "Transfer-Encoding" and k ~= "Connection" then + ngx.header[k] = v + end + end + ngx.say(res.body) +else + ngx.exit(ngx.HTTP_FORBIDDEN) +end + +httpc:close() -- 2.23.0

1 0

[PATCH v2 compass-ci] container/openresty-proxy-cache: add nginx.conf
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/nginx.conf | 85 ++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/openresty-proxy-cache/nginx.conf diff --git a/container/openresty-proxy-cache/nginx.conf b/container/openresty-proxy-cache/nginx.conf new file mode 100755 index 0000000..e3f11e0 --- /dev/null +++ b/container/openresty-proxy-cache/nginx.conf @@ -0,0 +1,85 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user nobody; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + proxy_cache_path /nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=10g + inactive=6000m use_temp_path=off; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + '"$upstream_cache_status" $body_bytes_sent'; + access_log /tmp/access.log main; + error_log /tmp/error.log; + + resolver 8.8.8.8; + client_max_body_size 700m; + server_tokens off; + + proxy_cache my_cache; + proxy_ignore_headers Set-Cookie Cache-Control Expires; + proxy_cache_valid 200 1440m; + proxy_cache_valid 301 302 5m; + proxy_set_header X-Real-Ip $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + add_header X-Cache $upstream_cache_status; + add_header X-Via $server_addr; + + server { + listen 443 ssl; + ssl_certificate /cert/ca.crt; + ssl_certificate_key /cert/ca.key; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + proxy_ssl_server_name on; + set_by_lua_block $cur_ups { + uri = ngx.var.request_uri + host = ngx.var.http_host + + path_file = "/srv/initrd/tar/"..uri + local f = io.open(path_file) + if f then + return "/127.0.0.1:443/tar/"..uri + end + return uri + } + proxy_pass $scheme:/$cur_ups; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } + server { + listen 80; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + + content_by_lua_file lua_script/forward.lua; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } +} -- 2.23.0

1 0

[PATCH lkp-tests] hosts/dc-*g: add nr_cpu
by Hu Xuejiao 08 Jan '21

08 Jan '21

[why] Because when job.rb(expand_params) deal with params/nr_threads [ ($1.to_f * nr_cpu / 100).to_i ], it will handle the exception --"rescue KeyError"-- in running docker [so] i need add the nr_cpu in dc-*g Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- hosts/dc-1g | 1 + hosts/dc-2g | 1 + hosts/dc-4g | 1 + hosts/dc-8g | 1 + 4 files changed, 4 insertions(+) diff --git a/hosts/dc-1g b/hosts/dc-1g index ec1e59b9..24ff4227 100644 --- a/hosts/dc-1g +++ b/hosts/dc-1g @@ -1 +1,2 @@ +nr_cpu: 1 memory: 1g diff --git a/hosts/dc-2g b/hosts/dc-2g index b9509a2c..82c2522e 100644 --- a/hosts/dc-2g +++ b/hosts/dc-2g @@ -1 +1,2 @@ +nr_cpu: 2 memory: 2g diff --git a/hosts/dc-4g b/hosts/dc-4g index 52e0636d..30353733 100644 --- a/hosts/dc-4g +++ b/hosts/dc-4g @@ -1 +1,2 @@ +nr_cpu: 4 memory: 4g diff --git a/hosts/dc-8g b/hosts/dc-8g index 333d6250..470f3094 100644 --- a/hosts/dc-8g +++ b/hosts/dc-8g @@ -1 +1,2 @@ +nr_cpu: 8 memory: 8g -- 2.23.0

1 0

[PATCH compass-ci] container/openresty-proxy-cache: add nginx.conf
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/nginx.conf | 85 ++++++++++++++++++++++ 1 file changed, 85 insertions(+) create mode 100755 container/openresty-proxy-cache/nginx.conf diff --git a/container/openresty-proxy-cache/nginx.conf b/container/openresty-proxy-cache/nginx.conf new file mode 100755 index 0000000..79aa10e --- /dev/null +++ b/container/openresty-proxy-cache/nginx.conf @@ -0,0 +1,85 @@ +# For more information on configuration, see: +# * Official English Documentation: http://nginx.org/en/docs/ + +user nobody; +worker_processes auto; +pid /run/nginx.pid; + +events {} + +http { + proxy_cache_path /nginx_cache levels=1:2 keys_zone=my_cache:10m max_size=10g + inactive=6000m use_temp_path=off; + + log_format main '$remote_addr - $remote_user [$time_local] "$request" ' + '$status "$http_referer" ' + '"$http_user_agent" "$http_x_forwarded_for" ' + '"$upstream_cache_status" $body_bytes_sent'; + access_log /tmp/access.log main; + error_log /tmp/error.log; + + resolver 8.8.8.8; + client_max_body_size 700m; + server_tokens off; + + proxy_cache my_cache; + proxy_ignore_headers Set-Cookie Cache-Control Expires; + proxy_cache_valid 200 1440m; + proxy_cache_valid 301 302 5m; + proxy_set_header X-Real-Ip $remote_addr; + proxy_set_header X-Forwarded-For $remote_addr; + add_header X-Cache $upstream_cache_status; + add_header X-Via $server_addr; + + server { + listen 443 ssl; + ssl_certificate /cert/ca.crt; + ssl_certificate_key /cert/ca.key; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + proxy_ssl_server_name on; + set_by_lua_block $cur_ups { + uri = ngx.var.request_uri + host = ngx.var.http_host + + path_file = "/srv/initrd/tar/"..uri + local f = io.open(path_file) + if f then + return "/127.0.0.1:443/tar/"..uri + end + return uri + } + proxy_pass $scheme:/$cur_ups; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } + server { + listen 80; + proxy_connect; + proxy_connect_allow 443; + + location / { + allow all; + + content_by_lua_file lua_script/forward.lua; + } + + location /tar { + allow all; + root /srv/initrd; + autoindex on; + autoindex_exact_size off; + } + + } +} -- 2.23.0

1 0

[PATCH compass-ci] container/openresty-proxy-cache: add start script
by cuiyili 08 Jan '21

08 Jan '21

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/openresty-proxy-cache/start | 23 +++++++++++++++++++++++ 1 file changed, 23 insertions(+) create mode 100755 container/openresty-proxy-cache/start diff --git a/container/openresty-proxy-cache/start b/container/openresty-proxy-cache/start new file mode 100755 index 0000000..b946426 --- /dev/null +++ b/container/openresty-proxy-cache/start @@ -0,0 +1,23 @@ +#!/bin/bash +# Refer to: https://hub.docker.com/r/openresty/openresty +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# SPDX-License-Identifier: MulanPSL-2.0+ + +. $CCI_SRC/container/defconfig.sh + +docker_rm openresty-proxy-cache + +cmd=( + docker run -d + --name openresty-proxy-cache + --restart=always + -p 11333:80 + -p 11334:443 + -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v $PWD/lua_script/forward.lua:/usr/local/openresty/nginx/lua_script/forward.lua + -v /srv/initrd/tar:/srv/initrd/tar:ro + -v /etc/localtime:/etc/localtime:ro + openresty:proxy_cache +) + +"${cmd[@]}" -- 2.23.0

1 0

[PATCH v5 compass-ci 7/7] send-internet-mail/start: add key REDIS_PORT
by Luan Shengde 08 Jan '21

08 Jan '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/send-internet-mail/start b/container/send-internet-mail/start index 02c6ab6..136435d 100755 --- a/container/send-internet-mail/start +++ b/container/send-internet-mail/start @@ -13,6 +13,7 @@ names = Set.new %w[ ROBOT_EMAIL_PASSWORD SEND_MAIL_PORT SENT_MAILDIR + REDIS_PORT ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH v5 compass-ci 6/7] lib/mail-post: add email checks
by Luan Shengde 08 Jan '21

08 Jan '21

add check email count only email addresses that email count within the limited value can be sent emails add check email mapping if has set email mapping for the entered address. the address will be converted to the mapping value Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 21 +++++++++++++++++++-- 1 file changed, 19 insertions(+), 2 deletions(-) mode change 100644 => 100755 lib/mail-post.rb diff --git a/lib/mail-post.rb b/lib/mail-post.rb old mode 100644 new mode 100755 index 9d3e601..803a134 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -57,14 +57,32 @@ post '/send_mail_encode' do send_mail(mail_info) end +def check_email_limit(mail_info) + email_limit = EmailRateLimit.new(mail_info) + email_limit.check_email_counts +end + +def check_email_mapping(mail_info) + email_mapping = EmailAddrMapping.new(mail_info) + email_mapping.check_email_mapping +end + def check_send_mail(mail_info) raise 'no/empty subject.' if mail_info['subject'].nil? || mail_info['subject'].empty? raise 'no/empty email_to address.' if mail_info['to'].nil? || mail_info['to'].empty? raise 'no/empty email content.' if mail_info['body'].nil? || mail_info['body'].empty? + + return mail_info unless ENV['SEND_MAIL_PORT'].to_s == '49000' + + mail_info = check_email_mapping(mail_info.clone) + mail_info = check_email_limit(mail_info.clone) + return mail_info end def send_mail(mail_info) - check_send_mail(mail_info) + mail_info = check_send_mail(mail_info) + return if mail_info['to'].empty? + mail = Mail.new do from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] @@ -79,7 +97,6 @@ def send_mail(mail_info) end def check_to_store_email(mail) - return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v5 compass-ci 5/7] send-internet-mail: email rate limit
by Luan Shengde 08 Jan '21

08 Jan '21

add email_in_limit to store email address that send count less than limit value add email_out_limit to store email address that send count beyond limit value the email queues email_in_limit/email_out_limit will be reset everyday at 00:00, every time send mail to an email address, the value for the email address in email_in_limit will +1, until the value up to the limited value, the email address will be moved to email_out_limit, and the email send to the address will be throw away. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../send-internet-mail/email_limit_queue.rb | 62 +++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100755 container/send-internet-mail/email_limit_queue.rb diff --git a/container/send-internet-mail/email_limit_queue.rb b/container/send-internet-mail/email_limit_queue.rb new file mode 100755 index 0000000..c7e3954 --- /dev/null +++ b/container/send-internet-mail/email_limit_queue.rb @@ -0,0 +1,62 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# check if the emails count has beyond the limit. +# email_in_limit is used to store email address email count +# less than the limit value. +# email_out_limit is used to store email address email count +# beyond the limit. +# case the email is the first one for the day, the email will +# be add to email_in_limit with value 1. +# case the email in email_in_limit but email count less than the +# limit value, its value will +1. +# case the email's email count up to the limit value, the email +# will be moved to email_out_limit. +# when the email is in email_out_limit, if send mail to the email +# address, the email will be kicked out from the email address bar. +class EmailRateLimit + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_counts + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + @mail_info['to'] = check_emails(email_to) + @mail_info['cc'] = check_emails(email_cc) + + return @mail_info + end + + def check_emails(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + if @redis.hexists 'email_out_limit', email + mail_list -= [email] + next + elsif @redis.hexists 'email_in_limit', email + email_account = @redis.hget 'email_in_limit', email + @redis.hset 'email_in_limit', email, email_account.to_i + 1 + else + @redis.hset 'email_in_limit', email, 1 + end + + change_queue(email) + end + return mail_list + end + + def change_queue(email) + return unless (@redis.hget 'email_in_limit', email).to_i >= 10 + + @redis.hdel 'email_in_limit', email + @redis.hset 'email_out_limit', email, 10 + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 4/7] send-internet-mail: add email mapping
by Luan Shengde 08 Jan '21

08 Jan '21

example: name => email_addr logo => email_addr email_addr_r => email_addr_d an email address can be set multi mappings, you can set a name or email to a specified email address when you write either a name or an email address in the send address bar, the email address will be converted to the specified address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_mapping.rb | 49 +++++++++++++++++++ 1 file changed, 49 insertions(+) create mode 100755 container/send-internet-mail/email_mapping.rb diff --git a/container/send-internet-mail/email_mapping.rb b/container/send-internet-mail/email_mapping.rb new file mode 100755 index 0000000..4c068b0 --- /dev/null +++ b/container/send-internet-mail/email_mapping.rb @@ -0,0 +1,49 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'redis' + +# email address translation according to the email_mapping queue +# email_mapping is used to set email mapping, example: +# email_mapping: +# name => email_address_d +# email_address_r1 => email_address_d +# email_address_r2 => email_address_d +# ... +# can set multi keys for an email address, the key can be a name, +# an email, or some logo else +# case add the key is added to email address bar, the key will be +# transferred to the destination email address +class EmailAddrMapping + def initialize(mail_info) + @mail_info = mail_info + @redis = Redis.new('host' => REDIS_HOST, 'port' => REDIS_PORT) + end + + def check_email_mapping + email_to = @mail_info['to'].clone + email_cc = @mail_info['cc'].clone + email_bcc = @mail_info['bcc'].clone + @mail_info['to'] = email_mapping(email_to) + @mail_info['cc'] = email_mapping(email_cc) + @mail_info['bcc'] = email_mapping(email_bcc) + + return @mail_info + end + + def email_mapping(mail_list) + return if mail_list.nil? || mail_list.empty? + + mail_list.clone.each do |email| + next unless @redis.hexists 'email_mapping', email + + mapped_email = @redis.hget 'email_mapping', email + mail_list -= [email] + mail_list << mapped_email unless mail_list.include? mapped_email + end + return mail_list + end +end -- 2.23.0

1 0

[PATCH v5 compass-ci 3/7] send-internet-mail: add timing work
by Luan Shengde 08 Jan '21

08 Jan '21

daily timing work: reset email rate limit queues Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/email_init.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100755 container/send-internet-mail/email_init.rb diff --git a/container/send-internet-mail/email_init.rb b/container/send-internet-mail/email_init.rb new file mode 100755 index 0000000..62f27ea --- /dev/null +++ b/container/send-internet-mail/email_init.rb @@ -0,0 +1,18 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'rufus-scheduler' +require 'redis' + +redis = Redis.new(host: REDIS_HOST, port: REDIS_PORT) +email_init = Rufus::Scheduler.new + +# Timing work for email counting everyday +# the email counts queues will be reset everyday +# to ensure can send mail in the new day to user +email_init.cron '0 0 * * *' do + redis.del 'email_in_limit' + redis.del 'email_out_limit' +end -- 2.23.0

1 0

[PATCH v5 compass-ci 2/7] send-internet-mail: optimize
by Luan Shengde 08 Jan '21

08 Jan '21

add init redis host/port add invoke code files Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/send-internet-mail.rb | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/container/send-internet-mail/send-internet-mail.rb b/container/send-internet-mail/send-internet-mail.rb index 8fdaa0d..7b357f8 100755 --- a/container/send-internet-mail/send-internet-mail.rb +++ b/container/send-internet-mail/send-internet-mail.rb @@ -6,7 +6,13 @@ require 'mail' require 'sinatra' -require "#{ENV['CCI_SRC']}/lib/mail-post" +REDIS_HOST = %x(/sbin/ip route | awk '/default/ {print $3}').chomp +REDIS_PORT = ENV['REDIS_PORT'] + +require_relative 'email_init' +require_relative 'email_limit_queue' +require_relative 'email_mapping' +require_relative '../../lib/mail-post' smtp = { address: 'smtp.qq.com', -- 2.23.0

1 0

[PATCH v5 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

1 0

[PATCH v4 compass-ci 1/7] send-internet-mail/Dockerfile: add install gem resources
by Luan Shengde 08 Jan '21

08 Jan '21

[why] redis is used to read/store email counts from/to redis queue rufus-scheduler is used to reset the queue as a timing task Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/send-internet-mail/Dockerfile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/send-internet-mail/Dockerfile b/container/send-internet-mail/Dockerfile index d151bda..03e3327 100644 --- a/container/send-internet-mail/Dockerfile +++ b/container/send-internet-mail/Dockerfile @@ -6,4 +6,4 @@ MAINTAINER shdluan(a)163.com ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y ruby-mail ruby-json ruby-sinatra + apt-get install -y ruby-mail ruby-json ruby-sinatra ruby-redis ruby-rufus-scheduler -- 2.23.0

2 2

review star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our review star, thanks Jihui! jimmy@crystal ~/todo/bin% ./review-star 7 Review email records in the last 7 days total review paragraph point people 53 31 68 26 Wu Fengguang 87 15 18 10 Wei Jihui 37 10 20 9 Lu Kaiyi 32 4 11 8 Yu Chuan 58 15 33 7 Luan Shengde 24 5 8 6 Xiao Shenwei 20 11 12 6 Xu Xijian 10 5 11 5 Wu Zhende 32 3 3 4 Hu Xuejiao 19 8 5 4 Liu Shaofei 22 9 9 4 Ren Wen 28 2 2 3 Cao Xueliang 29 6 5 2 Cui Yili 10 8 8 2 Wang Yong 17 2 3 2 Li Ping 3 3 5 2 Zhang Dewan 12 4 7 1 Lin Jiaxin 23 4 6 1 Wang Chenglong 10 2 2 1 Li Yuanchao 13 1 1 1 Lu Weitao 5 1 1 1 Zhang Yale 33 2 2 1 Liu Yinsi 6 1 2 0 Bai Jing

1 0

git count
by Wu Fengguang 08 Jan '21

08 Jan '21

Yu Chuan is our code star again, thanks Yu Chuan! 10 Yu Chuan 6 Wei Jihui 6 Wu Fengguang 6 Li Yuanchao 5 Liu Shaofei 4 Xu Xijian 4 Hu Xuejiao 4 Lu Weitao 4 Wu Zhende 4 Xiao Shenwei 4 Lu Kaiyi 3 Wang Chenglong 3 Li Ping 2 Bai Jing 2 Cao Xueliang 2 Cui Yili 2 Lin Jiaxin 1 Zhang Yale 1 Luan Shengde 0 Tong Qunfeng 0 Du Kaitian 0 Liu Yinsi 0 Sun Yukui 0 Zhang Yu 0 Wang Yong 0 Ren Wen 0 Zhang Dewan 0 Shi Zhichao

1 0

help star
by Wu Fengguang 08 Jan '21

08 Jan '21

Jihui is our help star, thanks Jihui! 15.2 Wu Fengguang 13.0 Wei Jihui 12.1 Yu Chuan 11.6 Xiao Shenwei 9.6 Lu Weitao 8.3 Ren Wen 7.5 Cao Xueliang 7.3 Wu Zhende 6.8 Zhang Yale 6.6 Wang Yong 6.1 Zhang Dewan 6.0 Liu Shaofei 4.8 Liu Yinsi 4.3 Xu Xijian 3.6 Wang Chenglong 3.3 Du Kaitian 2.8 Li Ping 2.7 Lin Jiaxin 2.3 Bai Jing 2.2 Cui Yili 2.0 Luan Shengde 1.3 Hu Xuejiao 0.7 Lu Kaiyi 0.0 Tong Qunfeng 0.0 Li Yuanchao 0.0 Wang Shuai 0.0 Cui Xucui

1 0

[PATCH lkp-tests] adaptation/openeuler: add dependency of pgbench
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- distro/adaptation/openeuler | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/adaptation/openeuler b/distro/adaptation/openeuler index 09120489..1938a8aa 100644 --- a/distro/adaptation/openeuler +++ b/distro/adaptation/openeuler @@ -295,3 +295,4 @@ ruby-full: ruby ruby-devel rubygem-rdoc linux-base: openssh-server: openssh openssh-client: openssh +postgresql-server: postgresql -- 2.23.0

2 1

[PATCH lkp-tests] tests/ebizzy: modify the value of iterations
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/ebizzy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/ebizzy b/tests/ebizzy index dcc301871..394bc09ba 100755 --- a/tests/ebizzy +++ b/tests/ebizzy @@ -19,7 +19,7 @@ duration=${duration%%s} [ -n "$mmap" ] && opt_mmap=" -m " [ -n "$linear" ] && opt_linear=" -z " - +[[ "$iterations" =~ ^[0-9]{3}x$ ]] && iterations=100 for i in $(seq 1 $iterations) do echo Iteration: $i -- 2.23.0

3 2

[PATCH v2 compass-ci 2/2] lib: add email_mapping_client
by Luan Shengde 08 Jan '21

08 Jan '21

used to - add email mapping - delete email mapping - re-map email mapping - search email mapping - delete email mapping queue(delete all email mappings) Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/email_mapping_client.rb | 47 +++++++++++++++++++++++++++++++++++++ 1 file changed, 47 insertions(+) create mode 100755 lib/email_mapping_client.rb diff --git a/lib/email_mapping_client.rb b/lib/email_mapping_client.rb new file mode 100755 index 0000000..002bf23 --- /dev/null +++ b/lib/email_mapping_client.rb @@ -0,0 +1,47 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class EmailMapping + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(r_name, d_email) + @r_name = r_name + @d_email = d_email + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_email_mapping + return false if @redis.hexists 'email_mapping', @r_name + + @redis.hset 'email_mapping', @r_name, @d_email + return true + end + + def delete_email_mapping + @redis.hdel 'email_mapping', @r_name + end + + def delete_mapping_queue + @redis.del 'email_mapping' + end + + def re_map_email + @redis.hset 'email_mapping', @r_name, @d_email + end + + def search_map + d_email = @redis.hget 'email_mapping', @r_name + + return d_email if d_email + + message = "#{@r_name} has not mapped an email yet." + return message + end +end -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin: add email-mapping
by Luan Shengde 08 Jan '21

08 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|logo appoint a name|email|logo to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like logo the file content consists of names and emails, like: name1: email_address1 name2: email_address2 you can map multi name to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 99 insertions(+) create mode 100755 sbin/email-mapping.rb diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..cfc70ec --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|logo', '--name name|email|logo', 'appoint a name|email|logo to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || nil + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 0

[PATCH compass-ci] uniform constant naming
by Lu Weitao 08 Jan '21

08 Jan '21

[Why] most file use "SEND_MAIL_HOST/PORT" [How] sed -i "s/[^_]MAIL_HOST/SEND_MAIL_HOST/g" `grep MAIL_HOST -rl compass-ci/*` sed -i "s/[^_]MAIL_PORT/SEND_MAIL_PORT/g" `grep MAIL_PORT -rl compass-ci/*` Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- container/delimiter/start | 4 ++-- container/extract-stats/start | 4 ++-- lib/constants.rb | 4 ++-- lib/mail_client.rb | 4 ++-- 4 files changed, 8 insertions(+), 8 deletions(-) diff --git a/container/delimiter/start b/container/delimiter/start index 5d7037d..cb44d84 100755 --- a/container/delimiter/start +++ b/container/delimiter/start @@ -9,8 +9,8 @@ require_relative '../defconfig.rb' names = Set.new %w[ ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT MONITOR_HOST MONITOR_PORT TASKQUEUE_HOST diff --git a/container/extract-stats/start b/container/extract-stats/start index 74cf917..6510564 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -12,8 +12,8 @@ names = Set.new %w[ REDIS_PORT ES_HOST ES_PORT - MAIL_HOST - MAIL_PORT + SEND_MAIL_HOST + SEND_MAIL_PORT LAB ] diff --git a/lib/constants.rb b/lib/constants.rb index c3c7543..a274adc 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -8,8 +8,8 @@ config = cci_defaults ES_HOST = config['ES_HOST'] || '172.17.0.1' ES_PORT = config['ES_PORT'] || 9200 -MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' -MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_HOST = config['SEND_MAIL_HOST'] || '172.17.0.1' +SEND_MAIL_PORT = config['SEND_MAIL_PORT'] || 49000 SRV_HTTP_HOST = config['SRV_HTTP_HOST'] || ENV['SRV_HTTP_HOST'] || '172.17.0.1' SRV_HTTP_PORT = config['SRV_HTTP_PORT'] || ENV['SRV_HTTP_PORT'] || 11300 diff --git a/lib/mail_client.rb b/lib/mail_client.rb index f63e65d..6f5a099 100644 --- a/lib/mail_client.rb +++ b/lib/mail_client.rb @@ -8,8 +8,8 @@ require_relative 'constants.rb' # mail client class class MailClient - HOST = (ENV.key?('MAIL_HOST') ? ENV['MAIL_HOST'] : MAIL_HOST) - PORT = (ENV.key?('MAIL_PORT') ? ENV['MAIL_PORT'] : MAIL_PORT).to_i + HOST = (ENV.key?('SEND_MAIL_HOST') ? ENV['SEND_MAIL_HOST'] : SEND_MAIL_HOST) + PORT = (ENV.key?('SEND_MAIL_PORT') ? ENV['SEND_MAIL_PORT'] : SEND_MAIL_PORT).to_i def initialize(host = HOST, port = PORT) @host = host @port = port -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/auto_submit.yaml: add/update linux repos
by Lin Jiaxin 08 Jan '21

08 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit.yaml | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 69b73ff..22172eb 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -4,7 +4,19 @@ c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml l/linux/linux: -- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/linux-next: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/openeuler-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/debian-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml +l/linux/archlinux-kernel: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

3 2

[PATCH lkp-tests] lib/bootstrap: fix failed to get MAC and IP
by Liu Yinsi 08 Jan '21

08 Jan '21

[error] LKP: HOSTNAME vm-8p64g.2288hv5-2s44p-384g--b5-3554212, MAC , IP , kernel 5.9.0-5-riscv64 1, serial console /dev/ttyS0 [ 75.489958] Kernel tests: Boot OK! ... [ 82.149060] sysrq: Resetting [why] check network ok or not by checking network adapter is not accurate, use ping ip more accurate. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/bootstrap.sh | 8 +------- 1 file changed, 1 insertion(+), 7 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index d4b41f3bd..b7795b40a 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -87,13 +87,7 @@ net_devices_link() network_ok() { - local i - for i in /sys/class/net/*/ - do - [ "${i#*/lo/}" != "$i" ] && continue - [ "$(cat $i/operstate)" = 'up' ] && return 0 - [ "$(cat $i/carrier 2>/dev/null)" = '1' ] && return 0 - done + ping -c 3 $LKP_SERVER | grep -q ttl && return 0 is_clearlinux && { net_devices_link up -- 2.23.0

2 2

[PATCH compass-ci 2/3] scheduler: remove prefix of 'kernel_uri' and 'initrds_uri'
by Ren Wen 08 Jan '21

08 Jan '21

Prefix 'kernel ' and 'initrd ' are just used by 'boot.ipxe'. Now need these uris without prefix for 'boot.libvirt', so remove them. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index e7b9e48..9f03a04 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -432,7 +432,7 @@ class Job end private def set_kernel_uri - self["kernel_uri"] = "kernel #{OS_HTTP_PREFIX}" + + self["kernel_uri"] = "#{OS_HTTP_PREFIX}" + JobHelper.service_path("#{linux_vmlinuz_path}") end @@ -496,7 +496,7 @@ class Job initrds_uri_values = [] of JSON::Any get_initrds().each do |initrd| - initrds_uri_values << JSON::Any.new("initrd #{initrd}") + initrds_uri_values << JSON::Any.new("#{initrd}") end @hash["initrds_uri"] = JSON::Any.new(initrds_uri_values) -- 2.23.0

1 0

[PATCH compass-ci 1/3] scheduler: add a new job field 'kernel_base_params'
by Ren Wen 08 Jan '21

08 Jan '21

Kernel params like 'initrd=xxx' is just for 'boot.ipxe'. Now need kernel params without 'initrd=xxx' for 'boot.libvirt', so add a job field 'kernel_base_params' for this. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 8 +------- src/scheduler/kernel_params.cr | 15 ++++++++++----- 2 files changed, 11 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 218e2e1..e7b9e48 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -489,13 +489,7 @@ class Job end private def initrds_basename - basenames = "" - - get_initrds().each do |initrd| - basenames += "initrd=#{File.basename(initrd)} " - end - - return basenames + return get_initrds().map { |initrd| "initrd=#{File.basename(initrd)}" } end private def set_initrds_uri diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index 76606aa..2319a9a 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -2,7 +2,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. class Job - private def kernel_common_params return "user=lkp job=/lkp/scheduled/job.yaml RESULT_ROOT=/result/job rootovl ip=dhcp ro" end @@ -15,10 +14,10 @@ class Job os_real_path = JobHelper.service_path("#{SRV_OS}/#{os_dir}") fs2root = { - "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path} #{initrds_basename()}", + "nfs" => "root=#{OS_HTTP_HOST}:#{os_real_path}", "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + - ",guest,ro,hard,vers=1.0,noacl,nouser_xattr #{initrds_basename()}", - "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0 #{initrds_basename()}", + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", + "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", "container" => "", } @@ -32,7 +31,13 @@ class Job private def set_kernel_params kernel_params_values = "#{kernel_common_params()} #{kernel_custom_params()} #{self.kernel_append_root} #{kernel_console()}" kernel_params_values = kernel_params_values.split(" ").map(&.strip()).reject!(&.empty?) + # *kernel_base_params* is for boot.libvirt + @hash["kernel_base_params"] = JSON.parse(kernel_params_values.to_json) + + _initrds_basenames = [] of String + _initrds_basenames = initrds_basename() unless os_mount == "container" + kernel_params_values += _initrds_basenames + # *kernel_params* is for boot.ipxe @hash["kernel_params"] = JSON.parse(kernel_params_values.to_json) end - end -- 2.23.0

1 0

[PATCH v6 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 08 Jan '21

08 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-riscv64/z9.289591 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-x86_64/z9.289594 /srv/result/host-info/2021-01-08/vm-8p64g/debian-sid-aarch64/z9.289571 [how] put public options like $kernel, $initrd, $smp together, set individual options like $machine, $bios, $nic according to different qemu command, then merge all options to run qemu. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 116 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 101 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..363cf4c 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,57 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + qemu-system-riscv64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +122,83 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} + +individual_option() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device $device + -netdev $netdev + ) + ;; + esac +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option +individual_option + run_qemu -- 2.23.0

1 0

[PATCH v3 compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..c9b5df4 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "$compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: modify the git download script
by Hu Xuejiao 08 Jan '21

08 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 42 ++++++++++++++++-------------------------- 1 file changed, 16 insertions(+), 26 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 48741ec..6174916 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -5,31 +5,21 @@ # fix files permissions cause container service start failed umask 002 -# download git trees - -cd /c || exit - -# git clone https://gitee.com/openeuler/compass-ci.git -# modify and manual run for now: -[ -d "compass-ci/.git" ] || { - git clone https://gitee.com/wu_fengguang/compass-ci.git - - cd /c/compass-ci || exit - [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" - - ln -s compass-ci /c/cci +download_repo() +{ + local repo="$1" + local commit_id="$2" + + [ -d "/c/$repo/.git" ] || { + git clone https://gitee.com/wu_fengguang/"$repo".git /c/"$repo" + + [ "$repo" == "compass-ci" ] && ln -s /c/"$repo" /c/cci + } + + cd /c/"$repo" || exit + [ -n "$commit_id" ] && git reset --hard "$commit_id" } -[ -d "lkp-tests/.git" ] || { - git clone https://gitee.com/wu_fengguang/lkp-tests.git - - cd /c/lkp-tests || exit - [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" -} - -[ -d "upstream-repos/.git" ] || { - git clone https://gitee.com/wu_fengguang/upstream-repos.git - - cd /c/upstream-repos || exit - [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" -} +download_repo "compass-ci" "compass_commit_id" +download_repo "lkp-tests" "$lkp_commit_id" +download_repo "upstream-repos" "$upstream_commit_id" -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] assign-account: add 'lab' to default yaml
by Luan Shengde 08 Jan '21

08 Jan '21

the default yaml file will contain: my_email my_name lab lab is used to read the lab yaml according to its value. example: if the value of lab is 'z9', the sched will read token from file: z9.yaml Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index af08d91..7e0a5a7 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -173,6 +173,7 @@ class AccountStorage # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] + account_yaml['lab'] = @data['lab'] File.open(default_yaml, 'w') do |f| f.puts account_yaml.to_yaml -- 2.23.0

2 1

[PATCH lkp-tests 1/2] lib/job.rb: add fault_reproduction
by Wei Jihui 08 Jan '21

08 Jan '21

do fault reproduction when submit a failed job Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- lib/job.rb | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/lib/job.rb b/lib/job.rb index 8346db87..631ebfc9 100755 --- a/lib/job.rb +++ b/lib/job.rb @@ -255,6 +255,7 @@ class Job hash.delete('override') end hash.merge!(@overrides) + hash.merge!(fault_reproduction) if hash.has_key?("id") @jobs.concat(multi_args(hash)) # return [hash] or [h1,h2] end rescue StandardError => e @@ -285,6 +286,10 @@ class Job @jobfile = jobfile end + def fault_reproduction + wait = load_yaml("#{lkp_src}/jobs/fault_reproduction.yaml", {}) + end + def multi_args(hash) jobs_array = [] hash.each { |key, value| -- 2.23.0

2 1

[PATCH lkp-tests] bootstrap: don't try to install ntpdate
by Wu Fengguang 08 Jan '21

08 Jan '21

It's not necessary for every boot. Let's not slow down every test run. Signed-off-by: Wu Fengguang <wfg(a)mail.ustc.edu.cn> --- lib/bootstrap.sh | 14 -------------- 1 file changed, 14 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index d4b41f3b..b04ea3dd 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -237,20 +237,6 @@ run_ntpdate() [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - has_cmd ntpdate || { - detect_system - local DISTRO=$_system_name_lowercase - [ "$DISTRO" == "unknown" ] && { - echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." - return - } - has_cmd ruby || { - echo "[WARNING] Cannot find ruby in $DISTRO, so skip run_ntpdate." - return - } - . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) - } - has_cmd ntpdate || return local hour="$(date +%H)" -- 2.23.0

1 0

[PATCH v3 lkp-tests] lib/bootstrap.sh: 'run-ipconfig' step should be before the check step
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] The `run-ipconfig` step is the IP configuration step for initramfs; The `[ -z "$net_devices"]` step is check whether the network is normal. Now the `check normal` step is before 'ip configuration' step, so we encounter the following error: + export NO_NETWORK=1 + echo export NO_NETWORK=1 due to no net devices + return [Debug Msg] + [ ! -e /usr/share/initramfs-tools/scripts/functions ] + get_net_devices + local net_devices + local i + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ /sys/class/net/enp0s1/ != /sys/class/net/enp0s1/ ] + [ p0s1/ != /sys/class/net/enp0s1/ ] + basename /sys/class/net/enp0s1/ + net_devices= enp0s1 + [ != /sys/class/net/lo/ ] + continue + echo enp0s1 + local net_devices= enp0s1 + [ -z ] + warn_no_eth0 + [ -f /proc/config.gz ] + return + echo ls /sys/class/net + ls /sys/class/net + is_virt + [ -n ] + [ -n ] + has_cmd virt-what + command -v virt-what + grep -q -w hypervisor /proc/cpuinfo + return 0 + export NO_NETWORK=1 + echo export NO_NETWORK=1 due to no net devices + return Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..770d550724f1 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -160,6 +160,9 @@ setup_network() return } + $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig + network_ok && return + local net_devices=$(get_net_devices) if [ -z "$net_devices" ]; then @@ -181,9 +184,6 @@ setup_network() fi fi - $LKP_DEBUG_PREFIX $LKP_SRC/bin/run-ipconfig - network_ok && return - local err_msg='IP-Config: Auto-configuration of network failed' dmesg | grep -q -F "$err_msg" || { # Include $err_msg in the error message so that it matches -- 2.23.0

2 3

[PATCH v3 lkp-tests] lib/bootstrap.sh: fix error when os is not integrated into lkp-tests
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] If the current os has not been integrated into lkp-tests, run method `detect_system`, then var `$_system_name_lowercase` will be 'unknown'. So we should adapt this. [Error] + local DISTRO=unknown ++ echo ntpdate ++ /home/yuchuan/lkp-tests/sbin/adapt-packages unknown Traceback (most recent call last): 3: from /home/yuchuan/lkp-tests/sbin/adapt-packages:11:in `<main>' 2: from /usr/share/ruby/psych.rb:497:in `load_file' 1: from /usr/share/ruby/psych.rb:497:in `open' /usr/share/ruby/psych.rb:497:in `initialize': No such file or directory @ rb_sysopen - /home/yuchuan/lkp-tests/distro/adaptation/unknown (Errno::ENOENT) [How] If the current os has not been integrated into lkp-tests, then return. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 7d15bb370fa9..912a76bb6687 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -240,6 +240,10 @@ run_ntpdate() has_cmd ntpdate || { detect_system local DISTRO=$_system_name_lowercase + [ "$DISTRO" == "unknown" ] && { + echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." + return + } . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) } -- 2.23.0

2 3

[PATCH v5 compass-ci] providers/qemu: adapt for x86_64 and riscv64
by Liu Yinsi 08 Jan '21

08 Jan '21

before: support kvm for aarch64 after: support kvm for aarch64, x86_64, riscv64 test: /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-riscv64/z9.288315 /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-x86_64/z9.288332 /srv/result/host-info/2021-01-07/vm-8p64g/debian-sid-aarch64/z9.288335 [how] put public options like $kernel, $initrd, $smp together, list different options like $machine, $bios, $nic according to different qemu command, then merge all options. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 110 ++++++++++++++++++++++++++++++++++++------ 1 file changed, 95 insertions(+), 15 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index c926d3d..a8a2b33 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,12 +18,6 @@ check_logfile() } } -qemu_command() -{ - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm -} - write_logfile() { ipxe_script=ipxe_script @@ -66,12 +60,56 @@ check_option_value() [ -n "$initrds" ] || exit } -get_initrd() +set_initrd() { initrd=initrd cat $initrds > $initrd } +set_bios() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + [ -f "$bios" ] || bios=/usr/share/ovmf/OVMF.fd +} + +set_helper() +{ + helper=/usr/libexec/qemu-bridge-helper + [ -f "$helper" ] || helper=/usr/lib/qemu/qemu-bridge-helper +} + +set_nic() +{ + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + +set_device() +{ + device="virtio-net-device,netdev=net0,mac=${mac}" +} + +set_netdev() +{ + netdev="bridge,br=br0,id=net0,helper=${helper}" +} + +set_qemu() +{ + qemu=( + qemu-system-aarch64 + qemu-kvm + qemu-system-x86_64 + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 +} + print_message() { echo $SCHED_PORT @@ -83,36 +121,78 @@ print_message() sleep 5 } -run_qemu() +public_option() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic -serial file:${log_file} -monitor null ) +} - "${kvm[@]}" --append "${append}" +run_qemu() +{ + case "$qemu" in + qemu-system-aarch64) + arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && arch_option=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + [ "$(arch)" == "x86_64" ] && arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-x86_64) + arch_option=( + -bios $bios + -nic $nic + ) + ;; + qemu-system-riscv64) + arch_option=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) + ;; + esac + + "${kvm[@]}" "${arch_option[@]}" --append "${append}" } check_logfile write_logfile -qemu_command parse_ipxe_script check_option_value -get_initrd + +set_initrd +set_bios +set_helper +set_nic +set_device +set_netdev +set_qemu print_message +public_option run_qemu -- 2.23.0

3 4

[PATCH lkp-tests] lib/bootstrap.sh: avoid the problem caused by the lack of ruby pkg
by Yu Chuan 08 Jan '21

08 Jan '21

[Why] We use $LKP_SRC/sbin/adapt-packages to find the ntpdate pkg suitable for different systems. But we are not sure that ruby is pre-installed in each system. So add a check, if ruby is not pre-installed, return. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 30e4af828bcd..d4b41f3bd0d5 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -244,6 +244,10 @@ run_ntpdate() echo "[WARNING] Current os cannot be recognized by lkp-test, so skip run_ntpdate." return } + has_cmd ruby || { + echo "[WARNING] Cannot find ruby in $DISTRO, so skip run_ntpdate." + return + } . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) } -- 2.23.0

2 1