- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] setup/pgbench-setup: start postgresql
by Hu Xuejiao 12 Jan '21

12 Jan '21

[why] when after installed postgresql-server, postgres(user) is already created, so initializing the postgresql directly. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- setup/pgbench-setup | 21 +++++---------------- 1 file changed, 5 insertions(+), 16 deletions(-) diff --git a/setup/pgbench-setup b/setup/pgbench-setup index cc0ce37da..ea55b2ebc 100755 --- a/setup/pgbench-setup +++ b/setup/pgbench-setup @@ -25,28 +25,17 @@ postgres_run_log() init_postgresql() { - grep -q postgres /etc/passwd && return - - useradd --home-dir /var/lib/postgresql --no-create-home --no-user-group --gid nogroup --system postgres - data_dir=/var/lib/postgresql/data + data_dir=/var/lib/pgsql/data mkdir -p "$data_dir" - chown postgres "$data_dir" - - bin_dir=/usr/lib/postgresql/*/bin - is_clearlinux && bin_dir=/usr/bin - postgres_run $bin_dir/initdb -D "$data_dir" 2>&1 - postgresql_conf_file="$data_dir"/postgresql.conf - run_dir=/var/run/postgresql - mkdir -p "$run_dir" - chown postgres "$run_dir" - postgres_run $bin_dir/pg_ctl -D "$data_dir" start + postgresql-setup initdb + service postgresql start } -postgresql_conf_file=$(find /etc/postgresql -name postgresql.conf | head -1) - init_postgresql +postgresql_conf_file=$(find "$data_dir" -name postgresql.conf | head -1) + [ -n "$max_wal_size" ] && log_eval "echo 'max_wal_size = $max_wal_size' >> $postgresql_conf_file" -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/auto_submit: only submit one branch job
by Liu Shaofei 12 Jan '21

12 Jan '21

[why] As some repositories have many branches, the git-mirror will submit jobs for each branch. So the large number of jobs are submitted for the same repository. To avoid this problem, just master branch is submitted. If repository does not have master branch, the last branch is selected to submit job. Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index 51ecc6a..950d1f0 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -28,16 +28,29 @@ class AutoSubmit return pkgbuild_repo end - def submit(newrefs_info, submit_argv) + # just submit master branch, if not master branch, submit last one branch. + def get_branch_argvs(newrefs_info) + repo = { 'branch' => '', 'commit' => '' } + newrefs_info['new_refs']['heads'].each do |branch, commit_id| - real_argvs = Array.new(submit_argv) - real_argvs.push("upstream_branch=#{branch.delete_prefix('refs/heads/')}") - real_argvs.push("upstream_commit=#{commit_id}") - tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{commit_id}) - real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + repo['branch'] = branch.delete_prefix('refs/heads/') + repo['commit'] = commit_id - system(real_argvs.join(' ')) + break if repo['branch'] == 'master' end + repo + end + + def submit(newrefs_info, submit_argv) + real_argvs = Array.new(submit_argv) + + repo = get_branch_argvs(newrefs_info) + + real_argvs.push("upstream_branch=#{repo['branch']}") + real_argvs.push("upstream_commit=#{repo['commit']}") + tag = %x(git -C /srv/git/#{newrefs_info['git_repo']}.git tag --points-at #{repo['commit']}) + real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + system(real_argvs.join(' ')) end def get_argvs(newrefs_info) -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 12 Jan '21

12 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/*: reduce parameters
by Wang Yong 12 Jan '21

12 Jan '21

Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- tests/build-pkg | 10 +++++----- tests/cci-depends | 11 +++++++---- tests/cci-makepkg | 11 +++++++---- 3 files changed, 19 insertions(+), 13 deletions(-) diff --git a/tests/build-pkg b/tests/build-pkg index 98341e7ac..31762384d 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -7,12 +7,12 @@ # - config # upstream_url -# We'll use the below global env vars, but not treat them as parameter to avoid +# We'll use the below global env vars, but not treat them as parameters to avoid # duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ -# - os -# - os_arch -# - os_version -# - os_mount +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, diff --git a/tests/cci-depends b/tests/cci-depends index 8baf023f3..c89737aca 100755 --- a/tests/cci-depends +++ b/tests/cci-depends @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/misc-base.sh diff --git a/tests/cci-makepkg b/tests/cci-makepkg index e737955b9..7d1e5cb15 100755 --- a/tests/cci-makepkg +++ b/tests/cci-makepkg @@ -1,9 +1,12 @@ #!/bin/bash # - benchmark -# - os_mount -# - os -# - os_arch -# - os_version + +# We'll use the below global env vars, but not treat them as parameters to avoid +# duplicates in result path like /openeuler-20.03-aarch64/openeuler-aarch64-cifs-2003/ +# os +# os_arch +# os_version +# os_mount ## makepkg is a script that automates the building of packages; it can download and validate source files, ## check dependencies, configure build-time settings, compile the sources, install into a temporary root, -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: change possible matches for webhook
by Li Yuanchao 12 Jan '21

12 Jan '21

As directory structure of upstream-repos changed, the possible git_repo of repository to match also need to change. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index f5c653b..2adaaa1 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -319,16 +319,12 @@ class MirrorMain end # example - # url: https://gitee.com/theprocess/oec-hardware git_repo: oec-hardware/oec-hardware # url: https://github.com/berkeley-abc/abc git_repo: a/abc/abc - # url: https://github.com/Siguyi/AvxToNeon git_repo: AvxToNeon/Siguyi + # url: https://github.com/Siguyi/AvxToNeon git_repo: a/AvxToNeon/Siguyi def get_git_repo(webhook_url) fork_name, project = webhook_url.split('/')[-2, 2] - git_repo = "#{project}/#{project}" - return git_repo if check_git_repo(git_repo, webhook_url) - - git_repo = "#{project}/#{fork_name}" + git_repo = "#{project[0].downcase}/#{project}/#{fork_name}" return git_repo if check_git_repo(git_repo, webhook_url) git_repo = "#{project[0].downcase}/#{project}/#{project}" -- 2.23.0

1 0

[PATCH v4 compass-ci 2/2] sbin: add email-mapping
by Luan Shengde 12 Jan '21

12 Jan '21

for user to: - add email mapping - delete email mapping - re-map email mapping - search email mapping from stdin or map file or - delete email mapping queue Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r] email_mapping -f mapping_file [-r] [-d] email_mapping --delete-queue options: -n, --name name|email|tag appoint a name|email|tag to add mapping -e, --email email_address appoint a email to be mapped -f, --file filename appoint a mapping file for name/email -r, --re-map do re-mappings -d, --delete delete email mappings --delete-queue delete mapping queue -s, --search search email mapping -h, --help show this message the name can be either a name, an email or something like a tag. the file content consists of names and emails, like: name: email_address_d tag: email_address_d email_address_r: email_address_d can map multi name|tag|email_address to a single email address Signed-off-by: Luan Shengde <shdluan(a)163.com> --- sbin/email-mapping.rb | 20 +++++++++++++------- 1 file changed, 13 insertions(+), 7 deletions(-) diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb index 9d72ecc..d3a9a62 100755 --- a/sbin/email-mapping.rb +++ b/sbin/email-mapping.rb @@ -9,7 +9,8 @@ require 'optparse' REDIS_HOST = 'localhost' REDIS_PORT = '6379' -require "#{ENV['CCI_SRC']}/lib/email_mapping_client" +# require_relative "#{ENV['CCI_SRC']}/lib/redis_client" +require '../lib/redis_client' r_name = nil d_email = nil @@ -74,17 +75,22 @@ end options.parse!(ARGV) def email_mapping(r_name, d_email, map_conf) - email_mapping = EmailMapping.new(r_name, d_email) + email_mapping = RedisClient.new('email_mapping') if map_conf['re_mapping'] - email_mapping.re_map_email + email_mapping.reset_hash_key(r_name, d_email) elsif map_conf['del_map'] - email_mapping.delete_email_mapping + email_mapping.delete_hash_key(r_name) elsif map_conf['search_map'] - puts email_mapping.search_map + mapped_email = email_mapping.search_hash_key(r_name) + if mapped_email.nil? || mapped_email.empty? + puts "#{r_name} has not been mapped an email yet." + else + puts mapped_email + end elsif map_conf['delete_queue'] - email_mapping.delete_mapping_queue + email_mapping.delete_queue else - return if email_mapping.add_email_mapping + return if email_mapping.add_hash_key(r_name, d_email) puts "#{r_name} has already mapped an email." end -- 2.23.0

1 1

[PATCH v4 compass-ci 1/2] lib: add redis_client
by Luan Shengde 12 Jan '21

12 Jan '21

used to: add redis hash key delete redis hash key reset redis hash key search redis hash key delete redis queue Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/redis_client.rb | 43 +++++++++++++++++++ sbin/email-mapping.rb | 99 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 142 insertions(+) create mode 100755 lib/redis_client.rb create mode 100755 sbin/email-mapping.rb diff --git a/lib/redis_client.rb b/lib/redis_client.rb new file mode 100755 index 0000000..142d7f0 --- /dev/null +++ b/lib/redis_client.rb @@ -0,0 +1,43 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'redis' +require 'json' + +# used to add/delete/re-map/search email mappings +class RedisClient + RDS_HOST = ENV['REDIS_HOST'] || REDIS_HOST + RDS_PORT = ENV['REDIS_PORT'] || REDIS_PORT + + def initialize(queue) + @queue = queue + @redis = Redis.new('host' => RDS_HOST, 'port' => RDS_PORT) + end + + def add_hash_key(key, value) + return false if @redis.hexists @queue, key + + @redis.hset @queue, key, value + return true + end + + def delete_hash_key(key) + @redis.hdel @queue, key + end + + def delete_queue + @redis.del @queue + end + + def reset_hash_key(key, value) + @redis.hset @queue, key, value + end + + def search_hash_key(key) + h_value = @redis.hget @queue, key + + return h_value if h_value + end +end diff --git a/sbin/email-mapping.rb b/sbin/email-mapping.rb new file mode 100755 index 0000000..9d72ecc --- /dev/null +++ b/sbin/email-mapping.rb @@ -0,0 +1,99 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' +require 'optparse' + +REDIS_HOST = 'localhost' +REDIS_PORT = '6379' + +require "#{ENV['CCI_SRC']}/lib/email_mapping_client" + +r_name = nil +d_email = nil +file_email_mapping = {} +map_conf = { + 're_mapping' => false, + 'del_map' => false, + 'search_map' => false, + 'delete_queue' => false +} + +options = OptionParser.new do |opts| + opts.banner = "Usage: email_mapping -n name [-e email_address] [-d] [-s] [-r]\n" + opts.banner += " email_mapping -f mapping_file [-r] [-d] [-s]\n" + opts.banner += " email_mapping --delete-queue\n" + + opts.separator '' + opts.separator 'options:' + + opts.on('-n name|email|tag', '--name name|email|tag', 'appoint a name|email|tag to add mapping') do |name| + r_name = name + end + + opts.on('-e email_address', '--email email_address', 'appoint a email to be mapped') do |email| + d_email = email + end + + opts.on('-f filename', '--file filename', 'appoint a mapping file for name/email') do |filename| + mapping_content = YAML.load_file(filename) || {} + file_email_mapping.update mapping_content + end + + opts.on('-r', '--re-map', 'do re-mappings') do + map_conf['re_mapping'] = true + end + + opts.on('-d', '--delete', 'delete email mappings') do + map_conf['del_map'] = true + end + + opts.on('--delete-queue', 'delete mapping queue') do + map_conf['delete_queue'] = true + end + + opts.on('-s', '--search', 'search email mapping') do + map_conf['search_map'] = true + end + + opts.on_tail('-h', '--help', 'show this message') do + puts opts + exit + end +end + +if ARGV.empty? + ARGV << '-h' +elsif (['-n', '-f', '--delete-queue'] - ARGV).eql? ['-n', '-f', '--delete-queue'] + ARGV.clear + ARGV << '-h' +end + +options.parse!(ARGV) + +def email_mapping(r_name, d_email, map_conf) + email_mapping = EmailMapping.new(r_name, d_email) + if map_conf['re_mapping'] + email_mapping.re_map_email + elsif map_conf['del_map'] + email_mapping.delete_email_mapping + elsif map_conf['search_map'] + puts email_mapping.search_map + elsif map_conf['delete_queue'] + email_mapping.delete_mapping_queue + else + return if email_mapping.add_email_mapping + + puts "#{r_name} has already mapped an email." + end +end + +if file_email_mapping.empty? + email_mapping(r_name, d_email, map_conf) +else + file_email_mapping.each do |name, email| + email_mapping(name, email, map_conf) + end +end -- 2.23.0

1 1

[PATCH v2 compass-ci 3/3] container/auto-submit: add build depends
by Liu Yinsi 12 Jan '21

12 Jan '21

store auto submit info to ES in build script, so add build depends. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build-depends | 1 + 1 file changed, 1 insertion(+) create mode 100644 container/auto-submit/build-depends diff --git a/container/auto-submit/build-depends b/container/auto-submit/build-depends new file mode 100644 index 0000000..8357fca --- /dev/null +++ b/container/auto-submit/build-depends @@ -0,0 +1 @@ +es -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] container/auto-submit: fix failed to submit job
by Liu Yinsi 12 Jan '21

12 Jan '21

[error] root@taishan200-2280-2s64p-256g--a9 /c/compass-ci/container/es# docker exec -it auto-submit sh /c/lkp-tests/sbin $ ./submit iperf.yaml testbox=vm-2p8g submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Missing required job key: 'my_email'. Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… [how] for user install compass-ci at their local server, store auto submit info to ES to pass verification when submit jobs in container. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/build | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/container/auto-submit/build b/container/auto-submit/build index 96383e3..9530d47 100755 --- a/container/auto-submit/build +++ b/container/auto-submit/build @@ -1,6 +1,14 @@ -#!/bin/sh +#!/usr/bin/env ruby # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true -docker build -t alpine:auto-submit . +require "#{ENV['CCI_SRC']}/lib/build_my_info_client" +config_yaml = '/etc/compass-ci/account/autosubmit.yaml' +option = YAML.load_file(config_yaml) + +build_my_info = BuildMyInfo.new(option['my_email'], option['my_name'], option['lab'], option['my_token']) +build_my_info.store_account_info + +system 'docker build -t alpine:auto-submit .' -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] container/auto-submit: bind mount volume
by Liu Yinsi 12 Jan '21

12 Jan '21

lkp-tests/lib/job.rb: load_self_config self_config_path = "#{ENV['HOME']}/.config/compass-ci" Dir.glob(['/etc/compass-ci/defaults/*.yaml', "#{self_config_path}/defaults/*.yaml"]).each do |file| [why] load_self_config support /etc/compass/defaults/ and $HOME/.config/compass-ci/defaults/, if mount /etc/compass-ci/account, it can't load config, so use /home/lkp/.config/compass-ci/defaults. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/auto-submit/start | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/auto-submit/start b/container/auto-submit/start index c25fbdd..e66cf57 100755 --- a/container/auto-submit/start +++ b/container/auto-submit/start @@ -21,7 +21,7 @@ cmd = %W[ -v #{ENV['LKP_SRC']}:#{DOCKER_LKP} -v /etc/localtime:/etc/localtime:ro -v /etc/compass-ci/defaults:/etc/compass-ci/defaults:ro - -v /etc/compass-ci/account:/etc/compass-ci/account:ro + -v /etc/compass-ci/account:/home/lkp/.config/compass-ci/defaults:ro -v /srv/git:/srv/git -w /c/compass-ci/sbin alpine:auto-submit -- 2.23.0

1 0