January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v3 lkp-tests] tests/*: reduce parameters
by Wang Yong 12 Jan '21

12 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 10 +++++----- tests/cci-depends | 11 +++++++---- tests/cci-makepkg | 11 +++++++---- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 98341e7ac..31762384d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,12 +7,12 @@ # - config # upstream_url -# We'll use the below global env vars, but not treat them as parameter to avoid +# We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -# - os -# - os_arch -# - os_version -# - os_mount +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..c89737aca 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..7d1e5cb15 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change possible matches for webhook
by Li Yuanchao 12 Jan '21

12 Jan '21

As directory structure of upstream-repos changed, the possible git_repo of repository to match also need to change. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index f5c653b..2adaaa1 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -319,16 +319,12 @@ class MirrorMain end # example - # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc - # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + # url: https://github.com/Siguyi/AvxToNeon git_repo: a/AvxToNeon/Siguyi def get_git_repo(webhook_url) fork_name, project = webhook_url.split('/')[-2, 2] - git_repo = "#{project}/#{project}" - return git_repo if check_git_repo(git_repo, webhook_url) - - git_repo = "#{project}/#{fork_name}" + git_repo = "#{project[0].downcase}/#{project}/#{fork_name}" return git_repo if check_git_repo(git_repo, webhook_url) git_repo = "#{project[0].downcase}/#{project}/#{project}" -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 12 Jan '21

12 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|tag appoint a name|email|tag to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like a tag. the file content consists of names and emails, like: name: email_address_d tag: email_address_d email_address_r: email_address_d can map multi name|tag|email_address to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb index 9d72ecc..d3a9a62 100755 --- a/sbin/email-mapping.rb +++ b/sbin/email-mapping.rb @@ -9,7 +9,8 @@ require 'optparse' REDIS_HOST = 'localhost' REDIS_PORT = '6379' -require "#{ENV['CCI_SRC']}/lib/email_mapping_client" +# require_relative "#{ENV['CCI_SRC']}/lib/redis_client" +require '../lib/redis_client' r_name = nil d_email = nil @@ -74,17 +75,22 @@ end options.parse!(ARGV) def email_mapping(r_name, d_email, map_conf) - email_mapping = EmailMapping.new(r_name, d_email) + email_mapping = RedisClient.new('email_mapping') if map_conf['re_mapping'] - email_mapping.re_map_email + email_mapping.reset_hash_key(r_name, d_email) elsif map_conf['del_map'] - email_mapping.delete_email_mapping + email_mapping.delete_hash_key(r_name) elsif map_conf['search_map'] - puts email_mapping.search_map + mapped_email = email_mapping.search_hash_key(r_name) + if mapped_email.nil? || mapped_email.empty? + puts "#{r_name} has not been mapped an email yet." + else + puts mapped_email + end elsif map_conf['delete_queue'] - email_mapping.delete_mapping_queue + email_mapping.delete_queue else - return if email_mapping.add_email_mapping + return if email_mapping.add_hash_key(r_name, d_email) puts "#{r_name} has already mapped an email." end -- 2.23.0

1 1

[PATCH v4 compass-ci 1/2] lib: add redis_client
by Luan Shengde 12 Jan '21

12 Jan '21

used to: add redis hash key delete redis hash key reset redis hash key search redis hash key delete redis queue Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/redis_client.rb | 43 +++++++++++++++++++ sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100755 lib/redis_client.rb create mode 100755 sbin/email-mapping.rb diff --git a/lib/redis_client.rb b/lib/redis_client.rb new file mode 100755 index 0000000..142d7f0 --- /dev/null +++ b/lib/redis_client.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class RedisClient + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(queue) + @queue = queue + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_hash_key(key, value) + return false if @redis.hexists @queue, key + + @redis.hset @queue, key, value + return true + end + + def delete_hash_key(key) + @redis.hdel @queue, key + end + + def delete_queue + @redis.del @queue + end + + def reset_hash_key(key, value) + @redis.hset @queue, key, value + end + + def search_hash_key(key) + h_value = @redis.hget @queue, key + + return h_value if h_value + end +end diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..9d72ecc --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 1

[PATCH v2 compass-ci 3/3] container/auto-submit: add build depends
by Liu Yinsi 12 Jan '21

12 Jan '21

store auto submit info to ES in build script, so add build depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends new file mode 100644 index 0000000..8357fca --- /dev/null +++ b/container/auto-submit/build-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/auto-submit: fix failed to submit job
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user install compass-ci at their local server, store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..9530d47 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -1,6 +1,14 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t alpine:auto-submit . +require "#{ENV['CCI_SRC']}/lib/build_my_info_client" +config_yaml = '/etc/compass-ci/account/autosubmit.yaml' +option = YAML.load_file(config_yaml) + +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) +build_my_info.store_account_info + +system 'docker build -t alpine:auto-submit .' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/auto-submit: bind mount volume
by Liu Yinsi 12 Jan '21

12 Jan '21

lkp-tests/lib/job.rb: load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" Dir.glob(['/etc/compass-ci/defaults/*.yaml', "#{self_config_path}/defaults/*.yaml"]).each do |file| [why] load_self_config support /etc/compass/defaults/ and $HOME/.config/compass-ci/defaults/, if mount /etc/compass-ci/account, it can't load config, so use /home/lkp/.config/compass-ci/defaults. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0

[PATCH v2 compass-ci] sbin/run_job: update the script
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] 1.host-info is adapted in physical machine, so replace yaml-file 2.after running multi-docker and multi-qemu, these processes need to be deleted Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/sbin/run_job b/sbin/run_job index 325fa9f..7405ca0 100755 --- a/sbin/run_job +++ b/sbin/run_job @@ -24,7 +24,7 @@ submit_one_job() submit_job() { - yaml_args=("$cci_makepkg_file" "host-info.yaml") + yaml_args=("$cci_makepkg_file" "ebizzy.yaml") for yaml in "${yaml_args[@]}" do submit_one_job & @@ -72,6 +72,18 @@ run_docker() done } +kill_all() +{ + for line in $(cat "$1") + do + [ -z "$ line" ] && break + kill -9 "$line" + done +} + run_qemu run_docker wait + +kill_all pid +kill_all $CCI_SRC/providers/dc.pid -- 2.23.0

2 1

[PATCH compass-ci] fix: the error docker name
by Wang Chenglong 12 Jan '21

12 Jan '21

[Error] + docker cp -a init_docker:/tmp/openeuler-20.09-20210112.cgz /home/wcl/ Error: No such container:path: init_docker:/tmp/openeuler-20.09-20210112.cgz --- container/osimage/openeuler/lib | 2 +- container/osimage/openeuler/run | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/container/osimage/openeuler/lib b/container/osimage/openeuler/lib index e0b7017..34dd0ec 100755 --- a/container/osimage/openeuler/lib +++ b/container/osimage/openeuler/lib @@ -40,6 +40,6 @@ load_docker_image() cp_image_to_host() { - docker cp -a init_docker:/tmp/${IMAGE_NAME} $HOME/ + docker cp -a initramfs_${image}:/tmp/${IMAGE_NAME} $HOME/ echo "result: $(ls $HOME/${IMAGE_NAME})" } diff --git a/container/osimage/openeuler/run b/container/osimage/openeuler/run index 5d7da72..6ed4069 100755 --- a/container/osimage/openeuler/run +++ b/container/osimage/openeuler/run @@ -24,5 +24,5 @@ cmd=( load_docker_image "${cmd[@]}" cp_image_to_host -docker_rm init_docker &> /dev/null +docker_rm initramfs_${image} &> /dev/null echo "build finished" -- 2.23.0

1 0

[PATCH compass-ci] lib/build_my_info_client: set default value
by Liu Yinsi 12 Jan '21

12 Jan '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/build_my_info_client.rb | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb index 162bf9f..c946bec 100755 --- a/lib/build_my_info_client.rb +++ b/lib/build_my_info_client.rb @@ -21,12 +21,13 @@ require_relative '../lib/es_client' # build_my_info = BuildMyInfo.new(email, name, lab) # build_my_info.config_my_info class BuildMyInfo - def initialize(my_email, my_name, lab) + def initialize(my_email, my_name, lab, my_token = nil) @lab = lab || 'nolab' + @my_token = my_token || %x(uuidgen).chomp @my_info = { 'my_email' => my_email, 'my_name' => my_name, - 'my_token' => %x(uuidgen).chomp, + 'my_token' => @my_token, 'my_login_name' => nil, 'my_commit_url' => nil, 'my_ssh_pubkey' => [] -- 2.23.0

1 0