November 2020 - Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci 3/4] assign-account: generate rsa keys
by Luan Shengde 13 Nov '20

13 Nov '20

generate rsa private/public key and return public key on jumper server [why] register the public key on testbox enable user one click to login in the testbox [how] generate rsa private/public key for user on jumper server return and register the public key to es Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 43 +++++++++++--------- 1 file changed, 24 insertions(+), 19 deletions(-) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index 137f5c6..9cf6044 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -34,12 +34,13 @@ setup_jumper_account_info read_jumper_info config_default_yaml config_authorized_key + generate_ssh_key the returned data for setup_jumper_account_info like: { - "account" => "guest", - "passwd" => "Use pub_key to login", - "jumper_ip" => "10.10.10.10", + "my_login_name" => "test_user", + "password" => "Use pub_key to login", + "jumper_host" => "0.0.0.0", "jumper_port" => "10000" } @@ -94,30 +95,35 @@ class AccountStorage end def setup_jumper_account_info - account_info = read_account_info - jumper_info = read_jumper_info - pub_key = @data['my_ssh_pubkey'] unless @data['my_ssh_pubkey'].nil? + login_name, password = read_account_info + jumper_host, jumper_port = read_jumper_info + pub_key = @data['my_ssh_pubkey'][0] - login_name = account_info[0] - password = if pub_key.nil? - account_info[1] - else - 'Use pub_key to login' - end + password = 'Use pub_key to login' unless pub_key.nil? + + ssh_dir = File.join('/home/', login_name, '.ssh') + config_authorized_key(login_name, pub_key, ssh_dir) unless pub_key.nil? + config_default_yaml(login_name) + my_jumper_pubkey = generate_ssh_key(login_name, ssh_dir) if @data['gen_sshkey'].eql? true jumper_account_info = { 'my_login_name' => login_name, 'my_password' => password, - 'jumper_host' => jumper_info[0].chomp, - 'jumper_port' => jumper_info[1].chomp + 'jumper_host' => jumper_host, + 'jumper_port' => jumper_port, + 'my_jumper_pubkey' => my_jumper_pubkey } - config_authorized_key(login_name, pub_key) unless pub_key.nil? - config_default_yaml(login_name) - return jumper_account_info end + def generate_ssh_key(login_name, ssh_dir) + Dir.mkdir ssh_dir, 0o700 unless File.exist? ssh_dir + %x(ssh-keygen -f "#{ssh_dir}/id_rsa" -N '' -C "#{login_name}@account-vm") + %x(chown -R #{login_name}:#{login_name} #{ssh_dir}) + File.read("/home/#{login_name}/.ssh/id_rsa.pub") + end + def config_default_yaml(login_name) default_yaml_dir = File.join('/home', login_name, '.config/compass-ci/defaults') FileUtils.mkdir_p default_yaml_dir @@ -132,8 +138,7 @@ class AccountStorage %x(chown -R #{login_name}:#{login_name} "/home/#{login_name}/.config") end - def config_authorized_key(login_name, pub_key) - ssh_dir = File.join('/home/', login_name, '.ssh') + def config_authorized_key(login_name, pub_key, ssh_dir) Dir.mkdir ssh_dir, 0o700 Dir.chdir ssh_dir f = File.new('authorized_keys', 'w') -- 2.23.0

2 1

[PATCH v2 lkp-tests] tats: add sysbench-mysql result parsing file
by Zhang Yu 13 Nov '20

13 Nov '20

[input] SQL statistics: queries performed: read: 208714422 write: 59632689 other: 29816345 total: 298163456 transactions: 14908172 (25024.26 per sec.) queries: 298163456 (496918.82 per sec.) ... [output] transactions: 25024.26 Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- stats/sysbench-mysql | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100755 stats/sysbench-mysql diff --git a/stats/sysbench-mysql b/stats/sysbench-mysql new file mode 100755 index 00000000..c7283126 --- /dev/null +++ b/stats/sysbench-mysql @@ -0,0 +1,34 @@ +#!/usr/bin/env ruby + +# SQL statistics: +# queries performed: +# read: 210212086 +# write: 60060591 +# other: 30030296 +# total: 300302973 +# transactions: 15015147 (25024.26 per sec.) +# queries: 300302973 (500485.16 per sec.) +# ignored errors: 2 (0.00 per sec.) +# reconnects:0 (0.00 per sec.) +# +# General statistics: +# total time: 600.0222s +# total number of events: 15015147 +# +# Latency (ms): +# min: 2.74 +# avg: 10.23 +# max: 2238.49 +# 95th percentile: 15.55 +# sum: 153573603.78 +# +# Threads fairness: +# events (avg/stddev): 58652.9180/4292.14 +# execution time (avg/stddev): 599.8969/0.01 + +while (line = STDIN.gets) + case line.chomp! + when /transactions:\s+\d+\s+\((\d+.?\d+)/ + puts "transactions: #{$1}" + end +end -- 2.23.0

3 5

Re: [PATCH v2 lkp-tests] tats: add sysbench-mysql result parsing file ^ stats
by Zhang Yale 13 Nov '20

13 Nov '20

Reply-To: In-Reply-To: <20201112101355.3076124-1-2134782174(a)qq.com> On Thu, Nov 12, 2020 at 06:13:55PM +0800, Zhang Yu wrote: >[input] > SQL statistics: > queries performed: > read: 208714422 > write: 59632689 > other: 29816345 > total: 298163456 > transactions: 14908172 (25024.26 per sec.) > queries: 298163456 (496918.82 per sec.) > ... > >[output] > transactions: 25024.26 > >Signed-off-by: Zhang Yu <2134782174(a)qq.com> >--- > stats/sysbench-mysql | 34 ++++++++++++++++++++++++++++++++++ > 1 file changed, 34 insertions(+) > create mode 100755 stats/sysbench-mysql > >diff --git a/stats/sysbench-mysql b/stats/sysbench-mysql >new file mode 100755 >index 00000000..c7283126 >--- /dev/null >+++ b/stats/sysbench-mysql >@@ -0,0 +1,34 @@ >+#!/usr/bin/env ruby >+ >+# SQL statistics: >+# queries performed: >+# read: 210212086 >+# write: 60060591 >+# other: 30030296 >+# total: 300302973 >+# transactions: 15015147 (25024.26 per sec.) >+# queries: 300302973 (500485.16 per sec.) >+# ignored errors: 2 (0.00 per sec.) >+# reconnects:0 (0.00 per sec.) >+# >+# General statistics: >+# total time: 600.0222s >+# total number of events: 15015147 >+# >+# Latency (ms): >+# min: 2.74 >+# avg: 10.23 >+# max: 2238.49 >+# 95th percentile: 15.55 >+# sum: 153573603.78 >+# >+# Threads fairness: >+# events (avg/stddev): 58652.9180/4292.14 >+# execution time (avg/stddev): 599.8969/0.01 >+ >+while (line = STDIN.gets) >+ case line.chomp! >+ when /transactions:\s+\d+\s+\((\d+.?\d+)/ >+ puts "transactions: #{$1}" >+ end >+end >-- >2.23.0 >

1 1

[PATCH v2 lkp-tests] stats: add hackbench-git result parsing script.
by Zhang Yale 13 Nov '20

13 Nov '20

[input] #Iterations: 1 2020-11-10 19:51:31 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.495 ... #Iterations: 10 2020-11-10 19:51:38 /lkp/benchmarks/hackbench-git/hackbench/hackbench 20 thread 1000 Running with 20*40 (== 800) tasks. Time: 0.512 [output] tasks: 800 Time: 0.5016 Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- stats/hackbench-git | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100755 stats/hackbench-git diff --git a/stats/hackbench-git b/stats/hackbench-git new file mode 100755 index 00000000..435948ea --- /dev/null +++ b/stats/hackbench-git @@ -0,0 +1,19 @@ +#!/usr/bin/env ruby + +LKP_SRC = ENV['LKP_SRC'] || File.dirname(File.dirname(File.realpath($PROGRAM_NAME))) + +require "#{LKP_SRC}/lib/statistics" + +time = [] +while (line = gets) + case line + when /^Running with .* $== (\d+)$ tasks/ + tasks = $1.to_i + when /^Time:/ + _name, seconds = line.split + time << seconds.to_f + end +end + +puts "tasks: #{tasks}" +puts "Time: #{time.sum / time.size}" -- 2.23.0

4 6

[PATCH compass-ci] remove actual ip in compass-ci code
by Xiao Shenwei 13 Nov '20

13 Nov '20

obtain the actual ip address from configuration file Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/scheduler/constants.cr | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/scheduler/constants.cr b/src/scheduler/constants.cr index 839c241..1b37333 100644 --- a/src/scheduler/constants.cr +++ b/src/scheduler/constants.cr @@ -15,10 +15,10 @@ LAB = (ENV.has_key?("LAB") ? ENV["LAB"] : "nolab") SCHED_HOST = (ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1") SCHED_PORT = (ENV.has_key?("SCHED_PORT") ? ENV["SCHED_PORT"] : 3000).to_i32 -INITRD_HTTP_HOST = (ENV.has_key?("INITRD_HTTP_HOST") ? ENV["INITRD_HTTP_HOST"] : "172.168.131.113") +INITRD_HTTP_HOST = (ENV.has_key?("INITRD_HTTP_HOST") ? ENV["INITRD_HTTP_HOST"] : "172.17.0.1") INITRD_HTTP_PORT = (ENV.has_key?("INITRD_HTTP_PORT") ? ENV["INITRD_HTTP_PORT"] : 8800).to_i32 -OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.168.131.113") +OS_HTTP_HOST = (ENV.has_key?("OS_HTTP_HOST") ? ENV["OS_HTTP_HOST"] : "172.17.0.1") OS_HTTP_PORT = (ENV.has_key?("OS_HTTP_PORT") ? ENV["OS_HTTP_PORT"] : 8000).to_i32 SRV_OS = "/srv/os" -- 2.23.0

1 0

[PATCH v5 compass-ci] src/lib/web_backend.rb: add date range limit feature
by Zhang Yuhang 13 Nov '20

13 Nov '20

[why] The "start_time" field need limit by date. example: GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 57 +++++++++++++++++-------------- src/lib/web_backend.rb | 15 ++++++++ 2 files changed, 46 insertions(+), 26 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index fcc765f..7db82ac 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -1,4 +1,6 @@ #!/usr/bin/ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'sinatra' @@ -12,56 +14,59 @@ set :show_exceptions, false set :environment, :production # GET /compare_candidates -# return: {"query_conditions": {...}, "dimension": [...]} +# Response: +# - {"query_conditions": {...}, "dimension": [...]} get '/compare_candidates' do compare_candidates end # GET /compare?suite=borrow&dimension=tbox_group -# query_conditions: -# > must +# must: +# - query_conditions # - os / os_version / suite / os_arch / tbox_group -# -# dimension: -# > must +# - dimension # - os / os_version / os_arch / suite / tbox_group # -# return: compare result string / "No Data." / "No Difference." +# Response: +# - compare result string / "No Data." / "No Difference." get '/compare' do compare(params) end -# GET /get_jobs?upstream_repo=AvxToNeon/AvxToNeon&page_size=12&page_num=3 -# upstream_repo: -# > optional -# -# page_size: -# > optional +# GET /get_jobs?upstream_repo=a&page_size=12&start_date=2020-11-03&end_date=2020-11-03 +# optional: +# - upstream_repo +# - regex search +# - os +# - os_version +# - os_arch +# - suite +# - category +# - testbox +# - job_state +# - id +# - start_date +# - end_date +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 jobs get '/get_jobs' do get_jobs(params) end # GET /get_repos?git_repo=a&page_size=12&page_num=3 -# git_repo: -# > optional -# -# page_size: -# > optional +# optional: +# - git_repo +# - page_size # - default: 20 -# -# page_num: -# > optional +# - page_num # - default: 1 # -# return: +# Response: # - default: 20 repos get '/get_repos' do get_repos(params) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 55c5e4a..b8df6f4 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -1,3 +1,5 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true require 'json' @@ -260,6 +262,17 @@ def get_jobs_result(result) jobs end +def get_job_query_range(condition_fields) + range = { start_time: {} } + start_date = condition_fields[:start_date] + end_date = condition_fields[:end_date] + + range[:start_time][:gte] = "#{start_date} 00:00:00" if start_date + range[:start_time][:lte] = "#{end_date} 23:59:59" if end_date + + { range: range } +end + def search_job(condition_fields, page_size, page_num) must = [] FIELDS.each do |field| @@ -272,6 +285,8 @@ def search_job(condition_fields, page_size, page_num) { term: { field => value } } end end + range = get_job_query_range(condition_fields) + must << range if range[:range][:start_time] result, total = es_search(must, page_size, page_num * page_size) return get_jobs_result(result), total end -- 2.23.0

1 0

[PATCH v4 compass-ci 3/3] fix: avoid to parse iscsiroot in setup-dracut.sh
by Xu Xijian 13 Nov '20

13 Nov '20

[why] When start with a new kernel, the dracut will parse iscsiroot and halted, which is unnecessary for us. [errmsg] [ 0.976307] dracut: FATAL: iscsiroot requested but kernel/initrd does not support iscsi [ 0.977011] dracut: Refusing to continue ... [ 1.063137] reboot: System halted Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 1 + 1 file changed, 1 insertion(+) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 2e676de..b9a7800 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -14,5 +14,6 @@ rm -rf /var/lib/apt/lists/* # Replace the runtime shell script with a custom shell script cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ +sed -i '/inst_hook cmdline 90 "$moddir\/parse-iscsiroot.sh"/d' /usr/lib/dracut/modules.d/95iscsi/module-setup.sh cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh -- 2.23.0

3 3

[PATCH v2 lkp-tests] depends: add nginx-server depends
by Wei Jihui 13 Nov '20

13 Nov '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx-server | 7 +++++++ 1 file changed, 7 insertions(+) create mode 100644 distro/depends/nginx-server diff --git a/distro/depends/nginx-server b/distro/depends/nginx-server new file mode 100644 index 00000000..da35d5f6 --- /dev/null +++ b/distro/depends/nginx-server @@ -0,0 +1,7 @@ +gcc +glibc +ethool +libssl-dev +zlib1g-dev +libpcre3-dev +libgoogle-perftools-dev -- 2.23.0

1 0

[PATCH v4 compass-ci] sbin/compare optimize help message
by Lu Weitao 13 Nov '20

13 Nov '20

[why] compare tool support output result with json | html format, but have no corresponding help message. [how] add the corresponding help message in --theme option [example] compare id=crystal.109482 id=crystal.109483 --theme=json Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/compare | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/compare b/sbin/compare index b13682e..c13ec1a 100755 --- a/sbin/compare +++ b/sbin/compare @@ -12,7 +12,7 @@ # compare "commit=a12d232e" "commit=b3bacc31" # compare "os=debian" "os=centos" -c "suite=iperf" # compare "os=centos" -d "os_version os_arch" -# compare "os=centos" -d "os_version os_arch" --color "classic" +# compare "os=centos" -d "os_version os_arch" --theme="classic" require 'optparse' require_relative '../lib/compare.rb' @@ -42,9 +42,9 @@ opt_parser = OptionParser.new do |opts| is_group = true end - opts.on('--color color', 'turn on colorful display with theme: classic|focus_good|focus_bad', - '|striking|light|none') do |color| - colorful = color + opts.on('--theme theme', 'turn on colorful display with theme: classic|focus_good|focus_bad', + '|striking|light|json|html') do |theme| + colorful = theme end opts.on('-t', '--template template', 'compare with user-defined template') do |t| -- 2.23.0

1 0

[PATCH v3 compass-ci] add the container/initrd-cifs to share /srv/initrd
by Xiao Shenwei 13 Nov '20

13 Nov '20

[why] our os-cifs container will mount three dir, /srv/os /srv/initrd /srv/result it's may cause: 1. our cifs service becomes slow 2. if the container exit, none of the three folders can be accessed. in order to reduce the coupling, need split them [how] one folder mount by one container [Usage]: Two ways for the client to mount initrd: 1. initrd. mount -t cifs -o guest,port=446 //ip/initrd /tmp/initrd 2. osimage. mount -t cifs -o guest,port=446 //ip/osimage /tmp/osimage Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- container/initrd-cifs/Dockerfile | 22 ++++++++++++++++++ container/initrd-cifs/build | 5 +++++ container/initrd-cifs/smb.conf | 38 ++++++++++++++++++++++++++++++++ container/initrd-cifs/start | 24 ++++++++++++++++++++ 4 files changed, 89 insertions(+) create mode 100644 container/initrd-cifs/Dockerfile create mode 100755 container/initrd-cifs/build create mode 100644 container/initrd-cifs/smb.conf create mode 100755 container/initrd-cifs/start diff --git a/container/initrd-cifs/Dockerfile b/container/initrd-cifs/Dockerfile new file mode 100644 index 0000000..341bfc9 --- /dev/null +++ b/container/initrd-cifs/Dockerfile @@ -0,0 +1,22 @@ +# Origin: https://github.com/Stanback/alpine-samba +# Copyright (C) 2016-2020 Eric D. Stanback +# SPDX-License-Identifier: GPL-3.0 + +FROM alpine:edge + +MAINTAINER Xiao Shenwei <xiaoshenwei96(a)163.com> + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories +RUN adduser -u 1090 -D lkp +RUN apk add --update \ + samba-common-tools \ + samba-client \ + samba-server \ + bash && \ + rm -rf /var/cache/apk/* + +COPY ./smb.conf /etc/samba/ + +EXPOSE 446/tcp + +ENTRYPOINT ["smbd", "--foreground", "--no-process-group", "--log-stdout"] diff --git a/container/initrd-cifs/build b/container/initrd-cifs/build new file mode 100755 index 0000000..ca9de39 --- /dev/null +++ b/container/initrd-cifs/build @@ -0,0 +1,5 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t alpine/initrd-cifs . diff --git a/container/initrd-cifs/smb.conf b/container/initrd-cifs/smb.conf new file mode 100644 index 0000000..eaddef7 --- /dev/null +++ b/container/initrd-cifs/smb.conf @@ -0,0 +1,38 @@ +# refer to https://lkml.org/lkml/2019/7/16/716 and https://lkml.org/lkml/2019/9/19/586 +[global] + workgroup = MYGROUP + server string = Samba Server + map to guest = Bad User + load printers = no + printing = bsd + printcap name = /dev/null + disable spoolss = yes + disable netbios = yes + server role = standalone + server services = -dns, -nbt + smb ports = 445 + create mode = 0777 + directory mode = 0777 + guest only = yes + guest ok = yes + server min protocol = NT1 + unix extensions = yes + mangled names = no + +[initrd] + path = /srv/initrd/ + comment = initrd + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp + +[osimage] + path = /srv/initrd/ + comment = osimage + browseable = yes + writable = yes + public = yes + force user = lkp + force group = lkp diff --git a/container/initrd-cifs/start b/container/initrd-cifs/start new file mode 100755 index 0000000..7232bdf --- /dev/null +++ b/container/initrd-cifs/start @@ -0,0 +1,24 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +. $CCI_SRC/container/defconfig.sh + +lsmod | grep -q "^cifs\s" || { + sudo modprobe cifs +} + +docker_rm initrd-cifs + +cmd=( + docker run + -d + -p 446:445 + -v /etc/localtime:/etc/localtime:ro + -v /srv/initrd:/srv/initrd + --name initrd-cifs + --restart=always + alpine/initrd-cifs +) + +"${cmd[@]}" -- 2.23.0

3 4