March 2021 - Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] sbin/submit: fix input docker_image value by cmdline
by Wei Jihui 09 Mar '21

09 Mar '21

[error] input: submit docker_image=centos:7 job.yaml output: docker_image: centos [why] submit split centos:7 to [centos, 7] [how] -s "docker_image: centos:7" ==> docker_image: centos:7 docker_image=centos:7 ==> docker_image: centos:7 Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- sbin/submit | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/sbin/submit b/sbin/submit index e5e41828..feb65d00 100755 --- a/sbin/submit +++ b/sbin/submit @@ -28,7 +28,7 @@ options = OptionParser.new do |opts| opts.separator 'options:' opts.on("-s 'KEY: VALUE'", "--set 'KEY: VALUE'", 'add YAML hash to job') do |key_value| - k, v = key_value.sub(' ', '').split(':') + k, v = key_value.sub(' ', '').split(':', 2) opt_set_key_value[k] = v end @@ -59,7 +59,7 @@ options = OptionParser.new do |opts| opts.on('-m', '--monitor', "monitor job status: use -m 'KEY: VALUE' to add rule") do opt_monitor = true - k, v = ARGV[0].sub(' ', '').split(':') if ARGV[0] + k, v = ARGV[0].sub(' ', '').split(':', 2) if ARGV[0] if (k && !k.empty?) && (v && !v.empty?) opt_monitor_query[k] = v ARGV.shift @@ -78,7 +78,7 @@ options.parse!(ARGV) seen_job_yaml = false ARGV.delete_if do |arg| if arg.index '=' - k, v = arg.sub(' ', '').sub(/=/, ':').split(':') + k, v = arg.sub(' ', '').split('=', 2) if seen_job_yaml opt_set_key_value[k] = v else -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow/install-tiny: call 5-build/register-account
by Liu Yinsi 09 Mar '21

09 Mar '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index 9e94e39..e48bd25 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -21,4 +21,5 @@ cd $CCI_SRC/sparrow || exit 4-docker/buildall 5-build/ipxe 5-build/os-ready +5-build/register-account 7-systemd/systemd-setup -- 2.23.0

1 0

[PATCH compass-ci 1/2] sparrow/install-tiny: call 0-package/read-config
by Liu Yinsi 09 Mar '21

09 Mar '21

for users locally deploy compass-ci, call 0-package/read-config, export my_name, my_email to register account. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/install-tiny | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/install-tiny b/sparrow/install-tiny index 5f026e5..9e94e39 100755 --- a/sparrow/install-tiny +++ b/sparrow/install-tiny @@ -7,6 +7,7 @@ [[ $CCI_SRC ]] || export CCI_SRC=$(cd $(dirname $(realpath $0)); git rev-parse --show-toplevel) cd $CCI_SRC/sparrow || exit +. 0-package/read-config 0-package/install 1-storage/tiny 1-storage/permission -- 2.23.0

1 0

[PATCH v3 lkp-tests] setup/simplify-ci: call $CCI_SRC/sparrow/install-client
by Liu Yinsi 09 Mar '21

09 Mar '21

[why] for simplicy-ci job test, if there is $SCHED_HOST, and export server_ip=$SCHED_HOST, when call $CCI_SRC/sparrow/install-client, config sparrow/setup.yaml and execute sparrow/0-package/read-config(export server_ip) is unnecessary. for locally deploy compass-ci client, must config sparrow/setup.yaml, and execute sparrow/0-package/read-config(export server_ip), if server_ip not exists, will call log_error() and exit. [how] in $LKP_SRC/setup/simplify-ci, just call $CCI_SRC/sparrow/install-client script, make code more reusablely. refer patch: Subject: [PATCH v5 compass-ci 1/2] sparrow/install-client: add deploy compass-ci client script Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- setup/simplify-ci | 61 ++++------------------------------------------- 1 file changed, 5 insertions(+), 56 deletions(-) diff --git a/setup/simplify-ci b/setup/simplify-ci index 5aede9cdb..83e20b5f8 100755 --- a/setup/simplify-ci +++ b/setup/simplify-ci @@ -2,8 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -: ${SCHED_HOST:=172.17.0.1} -: ${SCHED_PORT:=3000} +export server_ip=$SCHED_HOST git_ci() { @@ -17,61 +16,11 @@ git_ci() git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 } -dev_env() +deploy() { - export sched_host=$SCHED_HOST - export sched_port=$SCHED_PORT - 3-code/dev-env + cd /c/compass-ci/sparrow && ./install-client } -install_env() -{ - cd /c/compass-ci/sparrow || return - 0-package/install - 1-storage/tiny - 5-build/ipxe & - 1-storage/permission - 2-network/br0 - 2-network/iptables - 3-code/git - dev_env - . /etc/profile.d/compass.sh -} - -boot_ipxe() -{ - sed -i "s%172.17.0.1%$SCHED_HOST%g" /tftpboot/boot.ipxe - sed -i "s%3000%$SCHED_PORT%g" /tftpboot/boot.ipxe -} - -run_service() -{ - ( - cd $CCI_SRC/container/dnsmasq || return - ./build - ./start - boot_ipxe - )& - ( - cd $CCI_SRC/container/qemu-efi || return - ./build - ./install - )& - ( - cd $CCI_SRC/container/fluentd-base || return - ./build - cd $CCI_SRC/container/sub-fluentd || return - ./build - ./start - )& -} - -main() -{ - git_ci || return 1 - install_env - run_service -} +git_ci || exit 1 +deploy -main -wait -- 2.23.0

1 0

[PATCH lkp-tests] tests/rpmbuild-pkg: delete os_mount field in sync_dest dir.
by Li Ping 09 Mar '21

09 Mar '21

[why] rpm has no much to do with os_mount Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 38de16e66..b641e97d1 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -6,8 +6,7 @@ [ -n "$rpm_repo" ] || die "rpm_repo is empty" package_name=${rpm_repo##*/} -[[ "$os_mount" = "cifs" ]] && os_mount="nfs" -sync_dest="/initrd/rpmbuild-pkg/${os_mount}/${os}/${os_arch}/${os_version}/${package_name}" +sync_dest="/initrd/rpmbuild-pkg/${os}/${os_arch}/${os_version}/${package_name}" init_workspace() { -- 2.23.0

1 0

[PATCH v2 compass-ci] src/lib/job.cr: optimize the get_pkg_common_dir function
by Li Ping 09 Mar '21

09 Mar '21

[why] 1. rpm has no much to do with os_mount. add a judgment 2. optimize the code style Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 29 +++++++++++++++++------------ 1 file changed, 17 insertions(+), 12 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 0b4bee4..fe1b7c1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -258,23 +258,28 @@ class Job end def get_pkg_common_dir - pkg_style = nil + pkg_style = Hash(String, JSON::Any).new {|h, k| h[k] = JSON::Any.new(nil)} ["cci-makepkg", "cci-depends", "build-pkg", "rpmbuild-pkg"].each do |item| - pkg_style = @hash[item]? - break if pkg_style - end - return nil unless pkg_style + next unless @hash[item]? - pkg_style = JSON.parse("{}") if pkg_style == nil + pkg_style.merge!((a)hash[item].as_h) + break + end + return nil if pkg_style.empty? - tmp_os = pkg_style["os"]? == nil ? "#{os}" : pkg_style["os"] - tmp_os_arch = pkg_style["os_arch"]? == nil ? "#{os_arch}" : pkg_style["os_arch"] - tmp_os_version = pkg_style["os_version"]? == nil ? "#{os_version}" : pkg_style["os_version"] + tmp_os = pkg_style["os"].as_s? || "#{os}" + tmp_os_arch = pkg_style["os_arch"].as_s? || "#{os_arch}" + tmp_os_version = pkg_style["os_version"].as_s? || "#{os_version}" - tmp_os_mount = pkg_style["os_mount"]? == nil ? "#{os_mount}" : pkg_style["os_mount"] - mount_type = tmp_os_mount == "cifs" ? "nfs" : tmp_os_mount.dup + mount_type = pkg_style["os_mount"].as_s? || "#{os_mount}" + # same usage for client + mount_type = "nfs" if mount_type == "cifs" - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + if @hash["rpmbuild-pkg"]? + common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + else + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + end return common_dir end -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/2] sbin/submit: optimize submit -r
by Li Ping 09 Mar '21

09 Mar '21

[why] submit -m -r iperf.yaml submit /home/liping/lkp-tests/jobs/iperf.yaml, got job id=z9.1183160 submit /home/liping/lkp-tests/jobs/iperf.yaml, got job id=z9.1183161 query=>{"job_id":["z9.1183160","z9.1183161"]} connect to ws://172.168.131.2:11310/filter {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:36.649+0800","job_id":"z9.1183161","message":"","job_state":"submit","result_root":"/srv/result/iperf/2021-03-05/vm-2p8g/openeuler-20.03-aarch64/udp-300/z9.1183161","status_code":200,"method":"POST","resource":"/submit_job","elapsed":"24.68ms"} {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:39.699+0800","job_id":"z9.1183160","result_root":"/srv/result/iperf/2021-03-05/vm-2p8g/openeuler-20.03-aarch64/tcp-300/z9.1183160","job_state":"set result root","status_code":200,"method":"GET","resource":"/boot.ipxe/mac/0a-9f-90-36-ae-6b","elapsed":"8.75ms"} {"level_num":1,"level":"INFO","time":"2021-03-05T11:31:39.735+0800","job_id":"z9.1183160","job_state":"boot","status_code":200,"method":"GET","resource":"/boot.ipxe/mac/0a-9f-90-36-ae-6b","elapsed":"44.08ms"} ==> only get one result [how] get result_root directly from the msg not from fluentd data Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/monitor.rb | 36 ++++++++++-------------------------- sbin/monitor | 8 ++++++++ sbin/submit | 4 +++- 3 files changed, 21 insertions(+), 27 deletions(-) diff --git a/lib/monitor.rb b/lib/monitor.rb index 78582be6a..24191fa8d 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -11,7 +11,7 @@ require 'eventmachine' require 'json' class Monitor - attr_accessor :monitor_url, :query, :overrides, :action, :job, :result, :stop_query + attr_accessor :monitor_url, :query, :overrides, :action, :job, :result, :result_roots, :stop_query def initialize(monitor_url = '', query = {}, action = {}) @monitor_url = monitor_url @@ -25,9 +25,9 @@ class Monitor @exit_status_code = 0 @defaults = {} @result = [] + @result_roots = [] @stop_query = {} @reason = nil - @result_root = nil end def load_default @@ -95,29 +95,13 @@ class Monitor exec cmd end - def set_result_root(data) - return unless @action['mirror_result'] - return unless data['log'] - - data = JSON.parse(data['log']) - return unless data['result_root'] - - @result_root = data['result_root'] - end - - def lftp_mirror - @result_root.delete_prefix!('/srv') - srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' - srv_http_port = job['SRV_HTTP_PORT'] || '11300' - url = "http://#{srv_http_host}:#{srv_http_port}#{@result_root}" - system "lftp -c mirror #{url} >/dev/null 2>&1" - end - - def mirror_result(data) - if @result_root - lftp_mirror - else - set_result_root(data) + def mirror_result + @result_roots.each do |res| + res.to_s.delete_prefix!('/srv') + srv_http_host = job['SRV_HTTP_HOST'] || 'api.compass-ci.openeuler.org' + srv_http_port = job['SRV_HTTP_PORT'] || '11300' + url = "http://#{srv_http_host}:#{srv_http_port}#{res}" + system "lftp -c mirror #{url} >/dev/null 2>&1" end end @@ -163,7 +147,7 @@ class Monitor output(data) connect(data, ws) - mirror_result(data) + mirror_result stop(data, ws) if @action['stop'] end diff --git a/sbin/monitor b/sbin/monitor index def47bf6a..a2e677fc0 100755 --- a/sbin/monitor +++ b/sbin/monitor @@ -12,6 +12,7 @@ require 'yaml' opt_set_key_value = {} opt_action = {'output' => true} opt_job = {} +opt_results = [] opt_monitor_url = nil opt_timeout = 0 opt_stop_query = {} @@ -35,6 +36,12 @@ options = OptionParser.new do |opts| opt_action[action] = true end end + + opts.on('-r result_roots', '--results results', 'set monitor result_roots') do |result_roots| + result_roots.split(',').each do |res| + opt_results << res + end + end opts.separator ' action: output, output log' opts.separator ' action: connect, connect to the host when get ip' opts.separator ' action: stop, exit monitor client when get log' @@ -79,6 +86,7 @@ monitor = Monitor.new monitor.monitor_url = opt_monitor_url if opt_monitor_url monitor.overrides = opt_set_key_value monitor.action = opt_action +monitor.result_roots = opt_results monitor.job = opt_job monitor.stop_query = opt_stop_query diff --git a/sbin/submit b/sbin/submit index a6f0cd387..44fd0a7db 100755 --- a/sbin/submit +++ b/sbin/submit @@ -19,6 +19,7 @@ opt_monitor = false opt_monitor_query = {} opt_my_queue = false actions = ['output', 'stop'] +result_roots = [] options = OptionParser.new do |opts| opts.banner = 'Usage: submit [options] job1.yaml job2.yaml ...' @@ -197,6 +198,7 @@ ARGV.each do |jobfile| messages = scheduler_client.submit_job(job_json) JSON.parse(messages).each do |msg| if msg['message'].empty? + result_roots << msg['result_root'] job_ids << msg['job_id'].to_s puts("submit #{jobfile}, got job id=#{msg['job_id']}") else @@ -210,6 +212,6 @@ end if opt_monitor job_hash_list[0].delete('define_files') opt_monitor_query.merge!({'job_id' => job_ids}) - cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_hash_list[0].to_json}' -a #{actions.join(',')} -s 'job_state: extract_finished'" + cmd = "#{LKP_SRC}/sbin/monitor -f '#{opt_monitor_query.to_json}' -j '#{job_hash_list[0].to_json}' -a #{actions.join(',')} -r #{result_roots.join(',')} -s 'job_state: extract_finished'" exec cmd end -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: add my_email field
by Lu Weitao 09 Mar '21

09 Mar '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 9c9f7b6..edc6159 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -140,6 +140,9 @@ else "upstream_commit": { "type": "keyword" }, + "my_email": { + "type": "keyword" + }, "enqueue_time": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss" -- 2.23.0

1 0

[PATCH v6 compass-ci] doc/manual: add document for submit container
by Luan Shengde 09 Mar '21

09 Mar '21

Add English document for submit container Signed-off-by: Luan Shengde <shdluan(a)163.com> --- doc/manual/build-lkp-tests-container.en.md | 63 ++++++++++++++++++++++ 1 file changed, 63 insertions(+) create mode 100644 doc/manual/build-lkp-tests-container.en.md diff --git a/doc/manual/build-lkp-tests-container.en.md b/doc/manual/build-lkp-tests-container.en.md new file mode 100644 index 0000000..53f6887 --- /dev/null +++ b/doc/manual/build-lkp-tests-container.en.md @@ -0,0 +1,63 @@ +# Preface + +We provide a docker container to suit various of Linux OS(es). +In this case you do not need to install the lkp-tests to your local server. +Also you can avoid installation failures from undesired dependency package(s). + +# Prepare + + Do on your local server: + + - install docker + - apply account and config default yaml + - generate ssh key(s) + +# build container + +## 1. download resource + + Download lkp-tests and compass-ci to your local server. + + Command(s): + + git clone https://gitee.com/wu_fengguang/compass-ci.git + git clone https://gitee.com/wu_fengguang/lkp-tests.git + +## 2. add environment variable(s) + + Command(s): + + echo "export LKP_SRC=$PWD/lkp-tests" >> ~/.${SHELL##*/}rc + echo "export CCI_SRC=$PWD/compass-ci" >> ~/.${SHELL##*/}rc + source ~/.${SHELL##*/}rc + +## 3. build docker image + + Command(s): + + cd compass-ci/container/submit + ./build + +## 4. add executable file + + Command(s): + + ln -s $CCI_SRC/container/submit/submit /usr/bin/submit + +# try it + + instruction: + + You can directly use the command 'submit' to submit jobs. + It is the same as you install the lkp-tests on your own server. + It will start a disposable container to submit your job. + The container will attach the directory lkp-tests to the container itself. + You can edit the job yaml(s) in lkp-tests/jobs and it will take effect when you submit jobs. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml + + About submit: + + For detailed usage for command submit, please reference to: [submit user manual](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/su… -- 2.23.0

1 0

[PATCH compass-ci 2/2] container/mail-robot: fix nil error
by Luan Shengde 09 Mar '21

09 Mar '21

[why] When parsing user_info from the mail content, user_info may be nil if it's empty or a empty line. It may throw 'has no include? method for nil' error when use user_info.include? Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 272f1a1..e6c35f7 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -63,8 +63,10 @@ class ParseApplyAccountEmail users.delete('') users.each do |user| user_info = YAML.safe_load(user) + next if user_info.nil? + next unless user_info.include?('my_email') - users_info << user_info if user_info.include?('my_email') + users_info << user_info end return users_info -- 2.23.0

1 0