- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests: add ansible_tests script
by Zhang Yale 19 Jan '21

19 Jan '21

[ideology] 1. This script can test hundreds of the ansible roles in upstream git repo. 2. ansible roles are supported for the EL platform which has 11257 results in ansible-galaxy official website. 3. For testing ansible git repo, those cannot be tested in a unified way, I mainly test ansible roles git repo, and these roles repo will be more and more. [why] Command "ansible-playbook site.yml" can run in localhost, and only need to create an entry YAML file in which the value of hosts is localhost, for example "site.yml", created in the directory at the same level as the roles directory, which can invoke roles in it. And it dose not need to create a host file in $ansible_path. ``` site.yml --- - hosts: localhost roles: - roles: ansible-role-nginx ``` ansible-role-nginx which is an role in ~/.ansible/roles, can install from galaxy or git clone from github. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 94 +++++++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 39 deletions(-) diff --git a/tests/ansible_test b/tests/ansible_test index 09e5ca9e..193bd7e1 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -1,59 +1,75 @@ #!/bin/bash -# username -# rolename -# ansible_host -# ansible_user -# ansible_python_interpreter - -[ -n "$username" ] || echo "username is empty" -[ -n "$rolename" ] || echo "rolename is empty" -[ -n "$ansible_host" ] || $ansible_host=localhost -[ -n "$ansible_user" ] || $ansible_user=root -[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 - -build_mutual_trust() +# - ansible_repo + +set -x +. $LKP_SRC/lib/debug.sh + +check_vars() { - echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa - cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys - ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' + ansible_host=localhost + ansible_repo="jdauphant/ansible-role-ssl-certs" + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible } install_ansible() { - pip3 install ansible - ansible localhost -m ping - #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko - [ $? -eq 0 ] && echo "Ansible is successfully deployed!" + [[ "$os" == "openeuler" ]] && pip3 install ansible + [[ "$os" == "centos" ]] && yum install -y ansible + [[ "$os" == "debian" ]] && apt-get install -y ansible + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } } download_ansible_role() { - ansible-galaxy install $username.$rolename - [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone https://github.com/$ansible_repo.git || exit + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi } ansible_playbook_run() { - ansible_path=/root/.ansible - [ -d "$ansible_path" ] || mkdir -p /root/.ansible - cat <<-EOF > "$ansible_path"/hosts - $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter - EOF - [ $? -eq 0 ] && echo "Hostfile has been generated!" - - cat <<-EOF > "$ansible_path"/$rolename.yml + cat <<-EOF > "$ansible_path"/site.yml - hosts: $ansible_host - remote_user: $ansible_user roles: - - role: $username.$rolename + - role: $benchmark EOF - [ $? -eq 0 ] && echo "Playbook has been generated!" + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml - ansible-playbook "$ansible_path"/$rolename.yml - [ $? -eq 0 ] && echo "Your playbook is running successfully!" + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "playbook.run.fail: 1" && exit 1 +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run } -#build_mutual_trust -install_ansible -download_ansible_role -ansible_playbook_run +main -- 2.23.0

4 7

[PATCH compass-ci] answerback-email: optimize
by Luan Shengde 19 Jan '21

19 Jan '21

[why] the when define a constant, the constant may already defined in required files, when execute the script, my throw warnings: ./answerback-email.rb:32: warning: already initialized constant SEND_MAIL_HOST ~/compass-ci/lib/constants.rb:11: warning: previous definition of SEND_MAIL_HOST was here ./answerback-email.rb:33: warning: already initialized constant SEND_MAIL_PORT ~/compass-ci/lib/constants.rb:12: warning: previous definition of SEND_MAIL_PORT was here Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index a99306b..098561c 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -29,8 +29,8 @@ defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_HOST ||= defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 49000 LAB = defaults['lab'] TOOL_SERVER = 'z9' -- 2.23.0

2 2

[PATCH lkp-tests] tests/submit-cci: auto-submit testcase for other libraries
by Hu Xuejiao 19 Jan '21

19 Jan '21

[why] the file(testcase, job.yaml) of customer warehouse needs to be dragged to the lkp-tests to automatically submit the task. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/submit-cci | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 tests/submit-cci diff --git a/tests/submit-cci b/tests/submit-cci new file mode 100755 index 00000000..bfec6624 --- /dev/null +++ b/tests/submit-cci @@ -0,0 +1,44 @@ +#!/bin/bash + +gem install activesupport git rest-client elasticsearch + +config_variable() +{ + mkdir -p /etc/compass-ci/defaults + cat <<-EOF > /etc/compass-ci/defaults/$lab.yaml + SCHED_HOST: $SCHED_HOST + SCHED_PORT: $SCHED_PORT + my_email: $my_email + testbox: $testbox + EOF + + cat <<-EOF > /etc/profile.d/compass.sh + export PATH="$PATH:$LKP_SRC/sbin:$LKP_SRC/bin" + EOF + + . /etc/profile.d/compass.sh +} + +download_repo() +{ + if [ -n "$repo_path" ]; then + git clone git://"$LKP_SERVER"/"$repo_path" || exit 1 + git clone git://"$LKP_SERVER"/gitee.com/wu_fengguang/compass-ci.git + fi + + test_code=${repo##*/} + ruby compass-ci/sbin/build-my-info.rb -e "$my_email" -n "$USER" -l "$lab" + + cp -r "$test_code"/"$testcase" $LKP_SRC/tests + cp -r "$test_code"/"$job_yaml" $LKP_SRC/jobs +} + +submit_job() +{ + yaml=${job_yaml##*/} + submit "$yaml" +} + +config_variable +download_repo +submit_job -- 2.23.0

2 1

[PATCH v3 compass-ci] sbin/es-find support range query
by Lu Weitao 19 Jan '21

19 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..c57c02e 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,50 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + check_range_args(key, value) + range_hash[key] = assign_range(value) + end + + range_hash +end + +def check_range_args(key, value) + return if key && value + + warn "error: range value of \"#{key}\" missing", "tips: should give the input of -r like: + \"#{key}=2021-1-1 00:00:00,2021-1-1 23:59:59\" " + exit +end + +def assign_range(value) + range_value = {} + value_list = value.split(',') + if value_list.length > 1 + range_value[:gte] = value_list[0] + range_value[:lte] = value_list[1] + else + range_value[:gte] = value + end + + range_value +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +93,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +105,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

Re: [PATCH v3 compass-ci 3/3] compare_error_messages.rb/error_messages.rb: optimize string catenate
by Ren Wen 19 Jan '21

19 Jan '21

style: extra space. Confused subject. Thanks, RenWen Reply-To: In-Reply-To: <20210118114239.3343422-3-ljx.joe(a)qq.com> On Mon, Jan 18, 2021 at 07:42:39PM +0800, Lin Jiaxin wrote: >Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> >--- > lib/compare_error_messages.rb | 2 +- > lib/error_messages.rb | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > >diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb >index d9171bf..237b2c9 100644 >--- a/lib/compare_error_messages.rb >+++ b/lib/compare_error_messages.rb >@@ -73,7 +73,7 @@ def format_error_messages(error_messages, new_error_ids) > new_error_number = 0 > > error_messages.each do |k, v| >- if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) >+ if new_error_ids.include?("build-pkg.#{build_pkg_error_id(k)}") > new_error_number += 1 > formatted_error_messages = add_sign(formatted_error_messages, '>>', v) > else >diff --git a/lib/error_messages.rb b/lib/error_messages.rb >index d83d890..50f9637 100644 >--- a/lib/error_messages.rb >+++ b/lib/error_messages.rb >@@ -44,7 +44,7 @@ class ErrorMessages > error_messages_by_error_id = [] > error_messages = obtain_error_messages > error_messages.each do |k, v| >- if ('build-pkg.' + build_pkg_error_id(k)) == error_id >+ if "build-pkg.#{build_pkg_error_id(k)}" == error_id > error_messages_by_error_id += return_error_line ? [k] : v.to_a > end > end >-- >2.23.0 >

1 0

[PATCH v2 lkp-tests] jobs/submit-cci: add job.yaml
by Hu Xuejiao 19 Jan '21

19 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/submit-cci.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/submit-cci.yaml diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml new file mode 100644 index 00000000..e45bce72 --- /dev/null +++ b/jobs/submit-cci.yaml @@ -0,0 +1,4 @@ +suite: submit-cci +category: functional + +submit-cci: -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 19 Jan '21

19 Jan '21

Everytime git-mirror get new_refs or just git fetch, these actions and the time will be recorded in hash fork_stat{}. But when container git-mirror restart, fork_stat{} will be initialized. So historical data can't be kept. Now, git-mirror will get historical data from es when initialize hash fork_stat{}. So historical data will be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..1109cfe 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,31 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + fork_stat[:queued] = es_data['queued'] + fork_stat[:priority] = es_data['priority'] + fork_stat[:fetch_time] = es_data['fetch_time'] + fork_stat[:offset_fetch] = es_data['offset_fetch'] + fork_stat[:new_refs_time] = es_data['new_refs_time'] + fork_stat[:offset_new_refs] = es_data['offset_new_refs'] + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

1 0

[PATCH lkp-tests 2/2] error message enhances
by Zhang Dewan 19 Jan '21

19 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index ab1059a8..3e113093 100755 --- a/distro/debian +++ b/distro/debian @@ -62,7 +62,7 @@ install() # gawk_1%3a4.1.3+dfsg-0.1+b1_amd64.deb echo $(basename $deb_filename) >> opt/deb/keep-deb.$BM_NAME else - echo "error: package is not exist: $deb_pkgname " >&2 + echo "package is not exist: $deb_pkgname " >&2 return 1 fi done < opt/deb/sequence-deb.$BM_NAME -- 2.23.0

1 0

[PATCH lkp-tests 1/2] error message enhances
by Zhang Dewan 19 Jan '21

19 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index 77cd6258..ab1059a8 100755 --- a/distro/debian +++ b/distro/debian @@ -62,7 +62,7 @@ install() # gawk_1%3a4.1.3+dfsg-0.1+b1_amd64.deb echo $(basename $deb_filename) >> opt/deb/keep-deb.$BM_NAME else - echo "error: $deb_pkgname is not exist." >&2 + echo "error: package is not exist: $deb_pkgname " >&2 return 1 fi done < opt/deb/sequence-deb.$BM_NAME -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: firstly, `cat $initrds > $initrd`, secondly, `[ -n "$initrds" ] || exit` after: directly use `[ -n "$initrds" ] && cat $initrds > initrd || exit` Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0