- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests: add ansible_tests script
by Zhang Yale 19 Jan '21

19 Jan '21

[ideology] 1. This script can test hundreds of the ansible roles in upstream git repo. 2. ansible roles are supported for the EL platform which has 11257 results in ansible-galaxy official website. 3. For testing ansible git repo, those cannot be tested in a unified way, I mainly test ansible roles git repo, and these roles repo will be more and more. [why] Command "ansible-playbook site.yml" can run in localhost, and only need to create an entry YAML file in which the value of hosts is localhost, for example "site.yml", created in the directory at the same level as the roles directory, which can invoke roles in it. And it dose not need to create a host file in $ansible_path. ``` site.yml --- - hosts: localhost roles: - roles: ansible-role-nginx ``` ansible-role-nginx which is an role in ~/.ansible/roles, can install from galaxy or git clone from github. Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- tests/ansible_test | 94 +++++++++++++++++++++++++++------------------- 1 file changed, 55 insertions(+), 39 deletions(-) diff --git a/tests/ansible_test b/tests/ansible_test index 09e5ca9e..193bd7e1 100755 --- a/tests/ansible_test +++ b/tests/ansible_test @@ -1,59 +1,75 @@ #!/bin/bash -# username -# rolename -# ansible_host -# ansible_user -# ansible_python_interpreter - -[ -n "$username" ] || echo "username is empty" -[ -n "$rolename" ] || echo "rolename is empty" -[ -n "$ansible_host" ] || $ansible_host=localhost -[ -n "$ansible_user" ] || $ansible_user=root -[ -n "$ansible_python_interpreter" ] || $ansible_python_interpreter=/user/bin/python3 - -build_mutual_trust() +# - ansible_repo + +set -x +. $LKP_SRC/lib/debug.sh + +check_vars() { - echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /root/.ssh/id_rsa - cat /root/.ssh/id_rsa.pub > /root/.ssh/authorized_keys - ssh -o "StrictHostKeyChecking=no" -o "ExitOnForwardFailure=yes" -o "passwordAuthentication=no" root@$ansible__host 'chmod 600 ~/.ssh/authorized_keys' + ansible_host=localhost + ansible_repo="jdauphant/ansible-role-ssl-certs" + + [ -n "$ansible_repo" ] || die "ansible_repo is empty" + benchmark=${ansible_repo##*/} + + ansible_path=/root/.ansible + [ -d "$ansible_path" ] || mkdir -p /root/.ansible } install_ansible() { - pip3 install ansible - ansible localhost -m ping - #ansible all -m authorized_key -a "user=root key='{{ lookup('file', '/root/.ssh/id_rsa.pub') }}' path=/root/.ssh/authorized_keys manage_dir=no" --ask-pass -c paramiko - [ $? -eq 0 ] && echo "Ansible is successfully deployed!" + [[ "$os" == "openeuler" ]] && pip3 install ansible + [[ "$os" == "centos" ]] && yum install -y ansible + [[ "$os" == "debian" ]] && apt-get install -y ansible + + command -v ansible > /dev/null && { + ansible localhost -m ping + [ $? -eq 0 ] && { + echo "Ansible is successfully deployed!" + } + } } download_ansible_role() { - ansible-galaxy install $username.$rolename - [ $? -eq 0 ] && echo "Please check whether your rolename is correct!" + ansible_roles_path=$ansible_path/roles + [ -d $ansible_roles_path ] || mkdir -p $ansible_roles_path + cd $ansible_roles_path + git clone https://github.com/$ansible_repo.git || exit + # git clone git://$GIT_SERVER/$ansible_repo.git || exit + # ansible-galaxy install $username.$rolename + if [ $? -eq 0 ]; then + echo "$benchmark is successfully installed" + else + echo "Please check whether your ansible_repo is correct!" + fi } ansible_playbook_run() { - ansible_path=/root/.ansible - [ -d "$ansible_path" ] || mkdir -p /root/.ansible - cat <<-EOF > "$ansible_path"/hosts - $ansible_host ansible_user=$ansible_user ansible_python_interpreter=$ansible_python_interpreter - EOF - [ $? -eq 0 ] && echo "Hostfile has been generated!" - - cat <<-EOF > "$ansible_path"/$rolename.yml + cat <<-EOF > "$ansible_path"/site.yml - hosts: $ansible_host - remote_user: $ansible_user roles: - - role: $username.$rolename + - role: $benchmark EOF - [ $? -eq 0 ] && echo "Playbook has been generated!" + [ $? -eq 0 ] && { + echo "Playbook has been generated!" + } + + ansible-playbook "$ansible_path"/site.yml - ansible-playbook "$ansible_path"/$rolename.yml - [ $? -eq 0 ] && echo "Your playbook is running successfully!" + if [ $? -eq 0 ]; then + echo "Your playbook is running successfully!" + else + echo "playbook.run.fail: 1" && exit 1 +} + +main() +{ + check_vars + install_ansible + download_ansible_role + ansible_playbook_run } -#build_mutual_trust -install_ansible -download_ansible_role -ansible_playbook_run +main -- 2.23.0

4 7

[PATCH compass-ci] answerback-email: optimize
by Luan Shengde 19 Jan '21

19 Jan '21

[why] the when define a constant, the constant may already defined in required files, when execute the script, my throw warnings: ./answerback-email.rb:32: warning: already initialized constant SEND_MAIL_HOST ~/compass-ci/lib/constants.rb:11: warning: previous definition of SEND_MAIL_HOST was here ./answerback-email.rb:33: warning: already initialized constant SEND_MAIL_PORT ~/compass-ci/lib/constants.rb:12: warning: previous definition of SEND_MAIL_PORT was here Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index a99306b..098561c 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -29,8 +29,8 @@ defaults = relevant_defaults(names) JUMPER_HOST = defaults['JUMPER_HOST'] JUMPER_PORT = defaults['JUMPER_PORT'] || 29999 -SEND_MAIL_HOST = defaults['SEND_MAIL_HOST'] || 'localhost' -SEND_MAIL_PORT = defaults['SEND_MAIL_PORT'] || 49000 +SEND_MAIL_HOST ||= defaults['SEND_MAIL_HOST'] || 'localhost' +SEND_MAIL_PORT ||= defaults['SEND_MAIL_PORT'] || 49000 LAB = defaults['lab'] TOOL_SERVER = 'z9' -- 2.23.0

2 2

[PATCH lkp-tests] tests/submit-cci: auto-submit testcase for other libraries
by Hu Xuejiao 19 Jan '21

19 Jan '21

[why] the file(testcase, job.yaml) of customer warehouse needs to be dragged to the lkp-tests to automatically submit the task. Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/submit-cci | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) create mode 100755 tests/submit-cci diff --git a/tests/submit-cci b/tests/submit-cci new file mode 100755 index 00000000..bfec6624 --- /dev/null +++ b/tests/submit-cci @@ -0,0 +1,44 @@ +#!/bin/bash + +gem install activesupport git rest-client elasticsearch + +config_variable() +{ + mkdir -p /etc/compass-ci/defaults + cat <<-EOF > /etc/compass-ci/defaults/$lab.yaml + SCHED_HOST: $SCHED_HOST + SCHED_PORT: $SCHED_PORT + my_email: $my_email + testbox: $testbox + EOF + + cat <<-EOF > /etc/profile.d/compass.sh + export PATH="$PATH:$LKP_SRC/sbin:$LKP_SRC/bin" + EOF + + . /etc/profile.d/compass.sh +} + +download_repo() +{ + if [ -n "$repo_path" ]; then + git clone git://"$LKP_SERVER"/"$repo_path" || exit 1 + git clone git://"$LKP_SERVER"/gitee.com/wu_fengguang/compass-ci.git + fi + + test_code=${repo##*/} + ruby compass-ci/sbin/build-my-info.rb -e "$my_email" -n "$USER" -l "$lab" + + cp -r "$test_code"/"$testcase" $LKP_SRC/tests + cp -r "$test_code"/"$job_yaml" $LKP_SRC/jobs +} + +submit_job() +{ + yaml=${job_yaml##*/} + submit "$yaml" +} + +config_variable +download_repo +submit_job -- 2.23.0

2 1

[PATCH v3 compass-ci] sbin/es-find support range query
by Lu Weitao 19 Jan '21

19 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..c57c02e 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,50 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + check_range_args(key, value) + range_hash[key] = assign_range(value) + end + + range_hash +end + +def check_range_args(key, value) + return if key && value + + warn "error: range value of \"#{key}\" missing", "tips: should give the input of -r like: + \"#{key}=2021-1-1 00:00:00,2021-1-1 23:59:59\" " + exit +end + +def assign_range(value) + range_value = {} + value_list = value.split(',') + if value_list.length > 1 + range_value[:gte] = value_list[0] + range_value[:lte] = value_list[1] + else + range_value[:gte] = value + end + + range_value +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +93,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +105,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

Re: [PATCH v3 compass-ci 3/3] compare_error_messages.rb/error_messages.rb: optimize string catenate
by Ren Wen 19 Jan '21

19 Jan '21

style: extra space. Confused subject. Thanks, RenWen Reply-To: In-Reply-To: <20210118114239.3343422-3-ljx.joe(a)qq.com> On Mon, Jan 18, 2021 at 07:42:39PM +0800, Lin Jiaxin wrote: >Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> >--- > lib/compare_error_messages.rb | 2 +- > lib/error_messages.rb | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > >diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb >index d9171bf..237b2c9 100644 >--- a/lib/compare_error_messages.rb >+++ b/lib/compare_error_messages.rb >@@ -73,7 +73,7 @@ def format_error_messages(error_messages, new_error_ids) > new_error_number = 0 > > error_messages.each do |k, v| >- if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) >+ if new_error_ids.include?("build-pkg.#{build_pkg_error_id(k)}") > new_error_number += 1 > formatted_error_messages = add_sign(formatted_error_messages, '>>', v) > else >diff --git a/lib/error_messages.rb b/lib/error_messages.rb >index d83d890..50f9637 100644 >--- a/lib/error_messages.rb >+++ b/lib/error_messages.rb >@@ -44,7 +44,7 @@ class ErrorMessages > error_messages_by_error_id = [] > error_messages = obtain_error_messages > error_messages.each do |k, v| >- if ('build-pkg.' + build_pkg_error_id(k)) == error_id >+ if "build-pkg.#{build_pkg_error_id(k)}" == error_id > error_messages_by_error_id += return_error_line ? [k] : v.to_a > end > end >-- >2.23.0 >

1 0

[PATCH v2 lkp-tests] jobs/submit-cci: add job.yaml
by Hu Xuejiao 19 Jan '21

19 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/submit-cci.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/submit-cci.yaml diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml new file mode 100644 index 00000000..e45bce72 --- /dev/null +++ b/jobs/submit-cci.yaml @@ -0,0 +1,4 @@ +suite: submit-cci +category: functional + +submit-cci: -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 19 Jan '21

19 Jan '21

Everytime git-mirror get new_refs or just git fetch, these actions and the time will be recorded in hash fork_stat{}. But when container git-mirror restart, fork_stat{} will be initialized. So historical data can't be kept. Now, git-mirror will get historical data from es when initialize hash fork_stat{}. So historical data will be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..1109cfe 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,31 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + fork_stat[:queued] = es_data['queued'] + fork_stat[:priority] = es_data['priority'] + fork_stat[:fetch_time] = es_data['fetch_time'] + fork_stat[:offset_fetch] = es_data['offset_fetch'] + fork_stat[:new_refs_time] = es_data['new_refs_time'] + fork_stat[:offset_new_refs] = es_data['offset_new_refs'] + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

1 0

[PATCH lkp-tests 2/2] error message enhances
by Zhang Dewan 19 Jan '21

19 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index ab1059a8..3e113093 100755 --- a/distro/debian +++ b/distro/debian @@ -62,7 +62,7 @@ install() # gawk_1%3a4.1.3+dfsg-0.1+b1_amd64.deb echo $(basename $deb_filename) >> opt/deb/keep-deb.$BM_NAME else - echo "error: package is not exist: $deb_pkgname " >&2 + echo "package is not exist: $deb_pkgname " >&2 return 1 fi done < opt/deb/sequence-deb.$BM_NAME -- 2.23.0

1 0

[PATCH lkp-tests 1/2] error message enhances
by Zhang Dewan 19 Jan '21

19 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index 77cd6258..ab1059a8 100755 --- a/distro/debian +++ b/distro/debian @@ -62,7 +62,7 @@ install() # gawk_1%3a4.1.3+dfsg-0.1+b1_amd64.deb echo $(basename $deb_filename) >> opt/deb/keep-deb.$BM_NAME else - echo "error: $deb_pkgname is not exist." >&2 + echo "error: package is not exist: $deb_pkgname " >&2 return 1 fi done < opt/deb/sequence-deb.$BM_NAME -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: firstly, `cat $initrds > $initrd`, secondly, `[ -n "$initrds" ] || exit` after: directly use `[ -n "$initrds" ] && cat $initrds > initrd || exit` Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: [ -n "$initrds" ] || exit after: [ -n "$initrds" ] && cat $initrds > initrd || exit [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

2 4

[PATCH v3 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: firstly, `cat $initrds > $initrd`, secondly, `[ -n "$initrds" ] || exit` after: directly use `[ -n "$initrds" ] && cat $initrds > initrd || exit` [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 19 Jan '21

19 Jan '21

[why] start container os-cifs and os-nfs, need to use sudo command, so add it. container/os-cifs/start have follow line: sudo modprobe cifs container/os-nfs/start have follow line: sudo modprobe nfs Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/README.md: fix format error
by Liu Yinsi 19 Jan '21

19 Jan '21

1. fix format indentation. 2. add Link of debugging task, write PKGBUILD Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 40 +++++++++++++++++++++------------------- 1 file changed, 21 insertions(+), 19 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 89c43d8..794e433 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -4,24 +4,24 @@ ## 准备工作 - 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + >[openEuler 系统安装](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E… ### 操作指导 @@ -93,7 +93,7 @@ 1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… +2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add… 3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash @@ -107,9 +107,11 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… - 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) - 以查看iperf任务结果为例操如下： +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 + cat output ``` + + >**说明：** + >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: fix format error
by Liu Yinsi 19 Jan '21

19 Jan '21

1. fix format indentation. 2. add Link of debugging task, write PKGBUILD Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 45 ++++++++++++++++++++++++--------------------- 1 file changed, 24 insertions(+), 21 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index 89c43d8..c199397 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -4,24 +4,24 @@ ## 准备工作 - 硬件 - 服务器类型：ThaiShan200-2280 (建议) - 架构：aarch64 - 内存：>= 32GB - CPU：64 nuclear (建议) - 硬盘：>= 500G (建议划分独立分区) - + 服务器类型：ThaiShan200-2280 (建议) + 架构：aarch64 + 内存：>= 32GB + CPU：64 nuclear (建议) + 硬盘：>= 500G (建议划分独立分区) + >**说明：** - >划分较小独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… - >划分较大独立分区详细操作请参考[添加测试用例](https://gitee.com/wu_fengguang/compass-ci/blob/mast… + >[划分较小独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… + >[划分较大独立分区](https://gitee.com/wu_fengguang/compass-ci/blob/master/sparrow/1-s… - 软件 - OS：openEuler-aarch64-20.03 LTS - git：2.23.0版本 (建议) - 预留空间：>= 300G - 网络：可以访问互联网 - + OS：openEuler-aarch64-20.03 LTS + git：2.23.0版本 (建议) + 预留空间：>= 300G + 网络：可以访问互联网 + >**说明：** - >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… + >[openEuler 系统安装](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E… ### 操作指导 @@ -39,7 +39,7 @@ ```bash umask 002 && setenforce 0 ``` - + >**说明：** >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 @@ -70,7 +70,7 @@ ```bash submit iperf.yaml testbox=vm-2p8g ``` - + 执行上述命令正常情况下会提示信息如下: submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory @@ -93,7 +93,7 @@ 1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… -2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… +2. 根据测试需要[编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add…, [编写PKGBUILD](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… 3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… ```bash @@ -107,9 +107,12 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -2. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… - 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) - 以查看iperf任务结果为例操如下： +2. 在本地/srv/result/目录下根据测试用例名称/日期/[testbox](https://gitee.com/wu_fengguang/comp… + 以查看iperf任务结果为例操如下 ```bash - cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.03-aarch64/nolab.1 + cat output ``` + + >**说明：** + >[登陆测试环境调测任务](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manua… -- 2.23.0

1 1

[PATCH v2 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 19 Jan '21

19 Jan '21

container os-cifs and os-nfs will use sudo command, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

2 2

[PATCH compass-ci] sbin/auto_submit.yaml: update repos for build-pkg jobs
by Liu Shaofei 19 Jan '21

19 Jan '21

Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- sbin/auto_submit.yaml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 22172eb..c368ad6 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -19,5 +19,7 @@ l/linux/archlinux-kernel: - testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 build-linux.yaml - testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=cifs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 queue=vm-build-pkg build-pkg.yaml +- testbox=dc-8g os=openeuler os_version=20.03-pre os_mount=container os_arch=aarch64 docker_image=openeuler:20.03-pre queue=dc-build-pkg build-pkg.yaml +- testbox=dc-8g os=archlinux os_version=2021-01-06-pre os_mount=container os_arch=aarch64 docker_image=archlinux:2021-01-06-pre queue=dc-build-pkg build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 19 Jan '21

19 Jan '21

When init fork_stat{}, first query the es, get states in es if there is. So historical data can be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..1109cfe 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,31 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + fork_stat[:queued] = es_data['queued'] + fork_stat[:priority] = es_data['priority'] + fork_stat[:fetch_time] = es_data['fetch_time'] + fork_stat[:offset_fetch] = es_data['offset_fetch'] + fork_stat[:new_refs_time] = es_data['new_refs_time'] + fork_stat[:offset_new_refs] = es_data['offset_new_refs'] + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

2 1

[PATCH v2 compass-ci 1/2] qemu/kvm.sh: optimize the prompt when the error occurs
by Yu Chuan 19 Jan '21

19 Jan '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/log.sh | 32 ++++++++++++++++++++++++++++++++ providers/qemu/kvm.sh | 12 ++++++++---- 2 files changed, 40 insertions(+), 4 deletions(-) create mode 100644 lib/log.sh diff --git a/lib/log.sh b/lib/log.sh new file mode 100644 index 000000000000..56b08f77b3d0 --- /dev/null +++ b/lib/log.sh @@ -0,0 +1,32 @@ +#!/bin/bash + +die() +{ + log_error "$@" + exit 1 +} + +log() +{ + log_info "$@" +} + +log_info() +{ + echo "$(date +'%F %T') INFO -- $*" +} + +log_debug() +{ + echo "$(date +'%F %T') DEBUG -- $*" +} + +log_warn() +{ + echo "$(date +'%F %T') WARN -- $*" >&2 +} + +log_error() +{ + echo "$(date +'%F %T') ERROR -- $*" >&2 +} diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 35678fbd44ef..a136c8ff45cb 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -4,6 +4,8 @@ # - nr_cpu # - memory +source $CCI_SRC/lib/log.sh + : ${nr_cpu:=1} : ${memory:=1G} @@ -52,15 +54,17 @@ parse_ipxe_script() check_option_value() { - [ -s "$kernel" ] || { - echo "The kernel does not exist: $kernel" - exit + [ -n "$kernel" ] || { + log_info "Can not find job for current hostname: $hostname." + exit 0 } + + [ -s "$kernel" ] || die "Can not find kernel file or kernel file is empty: $kernel." # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit + [ -n "$initrds" ] || die "Can not find any initrd for current vm." } set_initrd() -- 2.23.0

2 2

[PATCH v3 lkp-tests] hosts: add new host file for libvirt
by Xiao Shenwei 19 Jan '21

19 Jan '21

[why] libvirt use a xml file to start vm <os> <kernel>xxx</kernel> <initrd>xxx</initrd> <cmdline>xxx</cmdline> </os> so, we can submit job use os_mount=initramfs to get kernel and initrd absolute path. Minimum memory limit: 16G Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/vt-2p16g | 5 +++++ hosts/vt-2p32g | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 hosts/vt-2p16g create mode 100644 hosts/vt-2p32g diff --git a/hosts/vt-2p16g b/hosts/vt-2p16g new file mode 100644 index 000000000..aecbb7948 --- /dev/null +++ b/hosts/vt-2p16g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G diff --git a/hosts/vt-2p32g b/hosts/vt-2p32g new file mode 100644 index 000000000..aa7c8c065 --- /dev/null +++ b/hosts/vt-2p32g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests 1/2] stats/crystal/rcuscale.cr: create rcuscale.cr as same content with rcuscale
by Lu Kaiyi 19 Jan '21

19 Jan '21

create rcuscale.cr as same content with rcuscale Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/rcuscale.cr | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) create mode 100644 stats/crystal/rcuscale.cr diff --git a/stats/crystal/rcuscale.cr b/stats/crystal/rcuscale.cr new file mode 100644 index 000000000..45c21f8cf --- /dev/null +++ b/stats/crystal/rcuscale.cr @@ -0,0 +1,30 @@ +#!/usr/bin/env ruby + +# Input example: +# [ 20.504916] rcu-scale:--- Start of test: nreaders=2 nwriters=2 verbose=1 shutdown=0 +# [ 20.521488] rcu-scale: rcu_scale_reader task started +# [ 20.526468] rcu-scale: rcu_scale_reader task started +# [ 20.536483] rcu-scale: rcu_scale_writer task started +# [ 20.541467] rcu-scale: rcu_scale_writer task started +# [ 31.728409] rcu-scale: rcu_scale_writer 0 has 100 measurements +# [ 31.730422] rcu-scale: rcu_scale_writer 1 has 100 measurements +# [ 31.744797] rcu-scale: Test complete +# [ 320.627980] rcu-scale: writer 0 gps: 101 +# [ 320.637100] rcu-scale: writer 1 gps: 100 +# [ 320.640206] rcu-scale: start: 30396186972 end: 31136206117 duration: 740019145 gps: 201 batches: 207 +# ... + +result = 'unknown' +type = 'unknown' + +while (line = STDIN.gets) + case line + when /^\[.*\] ([A-Za-z]+)-scale: Test complete/ + type = $1 + result = 'pass' + break + end +end + +stat = type + '.' + result +puts "#{stat}: 1" -- 2.23.0

2 1

[PATCH lkp-tests 2/2] stats/crystal/rcuscale.cr: convert rcuscale.cr to crystal with ruby2crystal tools
by Lu Kaiyi 19 Jan '21

19 Jan '21

/c/crystal/ruby_crystal_codemod/exe/ruby_crystal_codemod rcuscale.cr /c/lkp-tests/tools/ruby2crystal.sh rcuscale.cr Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- stats/crystal/rcuscale.cr | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/stats/crystal/rcuscale.cr b/stats/crystal/rcuscale.cr index 45c21f8cf..e5c0a2f28 100644 --- a/stats/crystal/rcuscale.cr +++ b/stats/crystal/rcuscale.cr @@ -1,4 +1,4 @@ -#!/usr/bin/env ruby +#!/usr/bin/env crystal # Input example: # [ 20.504916] rcu-scale:--- Start of test: nreaders=2 nwriters=2 verbose=1 shutdown=0 @@ -14,17 +14,17 @@ # [ 320.640206] rcu-scale: start: 30396186972 end: 31136206117 duration: 740019145 gps: 201 batches: 207 # ... -result = 'unknown' -type = 'unknown' +result = "unknown" +type = "unknown" while (line = STDIN.gets) case line when /^\[.*\] ([A-Za-z]+)-scale: Test complete/ type = $1 - result = 'pass' + result = "pass" break end end -stat = type + '.' + result +stat = type + "." + result puts "#{stat}: 1" -- 2.23.0

2 1

[PATCH v4 compass-ci] docker/docker.rb: add docker serial log
by Wu Zhende 19 Jan '21

19 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..56c9754 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_start_log(log_file, hash: {}) + start_time = Time.new + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time +end + +def record_end_log(log_file, start_time) + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,15 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + log_file = "#{LOG_DIR}/#{hostname}" + start_time = record_start_log(log_file, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_end_log(log_file, start_time) end def loop_main(hostname, queues) -- 2.23.0

1 0

[PATCH v3 compass-ci] docker/docker.rb: add docker serial log
by Wu Zhende 19 Jan '21

19 Jan '21

Generate serial log file. And add the start and end flags into it. It's for generate docker's dmsg file in result root. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index 8e489c6..edbc54d 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -18,6 +18,7 @@ names = Set.new %w[ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 +LOG_DIR = '/srv/cci/serial/logs' def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -82,12 +83,35 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path }, + { 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" + }, ENV['CCI_SRC'] + '/providers/docker/run.sh' ) clean_dir(load_path) end +def record_start_log(log_file, hash: {}) + start_time = Time.new + File.open(log_file, 'w') do |f| + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep(2) + f.puts "\n#{start_time.strftime('%Y-%m-%d %H:%M:%S')} starting DOCKER" + f.puts "\n#{hash['job']}" + end + return start_time +end + +def record_end_log(log_file, start_time) + duration = ((Time.new - start_time) / 60).round(2) + File.open(log_file, 'a') do |f| + f.puts "\nTotal DOCKER duration: #{duration} minutes" + end +end + def main(hostname, queues) set_host2queues(hostname, queues) url = get_url hostname @@ -95,10 +119,15 @@ def main(hostname, queues) hash = parse_response url return del_host2queues(hostname) if hash.nil? + log_file = "#{LOG_DIR}/#{hostname}" + start_time = record_start_log(log_file, hash: hash) + load_path = build_load_path(hostname) load_initrds(load_path, hash) start_container(hostname, load_path, hash) + del_host2queues(hostname) + record_end_log(log_file, start_time) end def loop_main(hostname, queues) -- 2.23.0

2 1

[PATCH lkp-tests] jobs/submit-cci.yaml: automatically submit test case.yaml
by Hu Xuejiao 19 Jan '21

19 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/submit-cci.yaml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 jobs/submit-cci.yaml diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml new file mode 100644 index 00000000..129ff44a --- /dev/null +++ b/jobs/submit-cci.yaml @@ -0,0 +1,3 @@ +suite: submit-cci +category: functional +submit-cci: -- 2.23.0

2 1

[PATCH v1 compass-ci] delimiter: use submit script in lkp-tests to submit job
by Cao Xueliang 19 Jan '21

19 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/constants.rb | 1 + src/delimiter/utils.rb | 17 +++++++++++------ 2 files changed, 12 insertions(+), 6 deletions(-) diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 23b5053..457e33d 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -13,3 +13,4 @@ DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" # The files which are in this dir can be uploaded by lkp-tests TMP_RESULT_ROOT = ENV['TMP_RESULT_ROOT'] || '/tmp/lkp/result' +PROCESS_JOB_YAML = "/tmp/process.yaml" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index aaad394..2c6549c 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -61,15 +61,17 @@ module Utils return monitor.run end + def save_job_to_yaml(job, yaml_file) + File.open(yaml_file, 'w') { |f| YAML.dump(job, f) } + end + def submit_job(job) - sched = SchedClient.new - response = sched.submit_job(job.to_json) + save_job_to_yaml(job, PROCESS_JOB_YAML) + response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" - res_arr = JSON.parse(response) - return nil if res_arr.empty? || !res_arr[0]['message'].empty? + return $1 if response =~ /job id=(.*)/ - # just consider build-pkg job - return res_arr[0]['job_id'] + return nil end # submit the bad job @@ -84,6 +86,8 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? + raise "the job is incredible for bisect: #{new_job_id}" unless credible?(job['bad_job_id'], new_job_id, error_id) + stats = query_stats(new_job_id, 10) raise "es cant query #{new_job_id} stats field!" unless stats @@ -138,6 +142,7 @@ module Utils job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id + job.delete('tboxgroup') job.delete('subqueue') job.delete('queue') job.delete('id') -- 2.23.0

1 0

[PATCH compass-ci] delimiter: use submit script in lkp-tests to submit job
by Cao Xueliang 19 Jan '21

19 Jan '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- providers/docker/run.sh | 2 ++ src/delimiter/constants.rb | 1 + src/delimiter/utils.rb | 17 +++++++++++------ 3 files changed, 14 insertions(+), 6 deletions(-) diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 4d53db7..5e692e4 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -26,9 +26,11 @@ cmd=( -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -e CCI_SRC=/c/compass-ci + -e LKP_SRC=/c/lkp-tests -v ${load_path}/lkp:/lkp -v ${DIR}/bin:/root/sbin:ro -v $CCI_SRC:/c/compass-ci:ro + -v $LKP_SRC:/c/lkp-tests:ro -v /srv/git:/srv/git:ro -v /srv/result:/srv/result:ro -v /etc/localtime:/etc/localtime:ro diff --git a/src/delimiter/constants.rb b/src/delimiter/constants.rb index 23b5053..457e33d 100644 --- a/src/delimiter/constants.rb +++ b/src/delimiter/constants.rb @@ -13,3 +13,4 @@ DELIMITER_TASK_QUEUE = 'delimiter' BISECT_RUN_SCRIPT = "#{ENV['CCI_SRC']}/src/delimiter/find-commit/bisect_run_script.rb" # The files which are in this dir can be uploaded by lkp-tests TMP_RESULT_ROOT = ENV['TMP_RESULT_ROOT'] || '/tmp/lkp/result' +PROCESS_JOB_YAML = "/tmp/process.yaml" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index aaad394..2c6549c 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -61,15 +61,17 @@ module Utils return monitor.run end + def save_job_to_yaml(job, yaml_file) + File.open(yaml_file, 'w') { |f| YAML.dump(job, f) } + end + def submit_job(job) - sched = SchedClient.new - response = sched.submit_job(job.to_json) + save_job_to_yaml(job, PROCESS_JOB_YAML) + response = %x(#{LKP_SRC}/sbin/submit #{PROCESS_JOB_YAML}) puts "submit job response: #{response}" - res_arr = JSON.parse(response) - return nil if res_arr.empty? || !res_arr[0]['message'].empty? + return $1 if response =~ /job id=(.*)/ - # just consider build-pkg job - return res_arr[0]['job_id'] + return nil end # submit the bad job @@ -84,6 +86,8 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? + raise "the job is incredible for bisect: #{new_job_id}" unless credible?(job['bad_job_id'], new_job_id, error_id) + stats = query_stats(new_job_id, 10) raise "es cant query #{new_job_id} stats field!" unless stats @@ -138,6 +142,7 @@ module Utils job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id + job.delete('tboxgroup') job.delete('subqueue') job.delete('queue') job.delete('id') -- 2.23.0

2 2

Re: [PATCH lkp-tests] jobs/submit-cci.yaml: automatically submit test case.yaml
by Zhang Yale 19 Jan '21

19 Jan '21

case.yaml? Reply-To: In-Reply-To: <20210119021655.3324160-1-1034502035(a)qq.com> On Tue, Jan 19, 2021 at 10:16:55AM +0800, Hu Xuejiao wrote: >Signed-off-by: Hu XueJiao <1034502035(a)qq.com> >--- > jobs/submit-cci.yaml | 3 +++ > 1 file changed, 3 insertions(+) > create mode 100644 jobs/submit-cci.yaml > >diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml >new file mode 100644 >index 00000000..129ff44a >--- /dev/null >+++ b/jobs/submit-cci.yaml >@@ -0,0 +1,3 @@ >+suite: submit-cci >+category: functional Here need a blank line Thanks, Yale >+submit-cci: >-- >2.23.0 >

1 0

[PATCH lab-z9] add 2288hv5-2s44p-384g--b2 host info
by Zhang Yale 19 Jan '21

19 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- hosts/2288hv5-2s44p-384g--b2 | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 hosts/2288hv5-2s44p-384g--b2 diff --git a/hosts/2288hv5-2s44p-384g--b2 b/hosts/2288hv5-2s44p-384g--b2 new file mode 100644 index 0000000..07e8a58 --- /dev/null +++ b/hosts/2288hv5-2s44p-384g--b2 @@ -0,0 +1,13 @@ +memory: 384G +arch: x86_64 +nr_node: 2 +nr_cpu: 88 +nr_ssd_partitions: 1 +ssd_partitions: + - /dev/disk/by-id/ata-MTFDDAK480TDC-1AT1ZABYY_20032653CD3C-part1 + - /dev/disk/by-id/ata-MTFDDAK480TDC-1AT1ZABYY_20032653CD3C-part2 +mac_addr: + - 6c:eb:b6:50:c1:b7 + - 6c:eb:b6:50:c1:b8 +model_name: Intel(R) Xeon(R) Gold 6266C CPU @ 3.00GHz +ipmi_ip: 9.3.3.7 -- 2.23.0

3 2

[PATCH lkp-tests] tests/deploy-cci: mount rootfs
by Liu Yinsi 19 Jan '21

19 Jan '21

[why] run qemu test needs os rootfs, so mount /srv/os and /srv/initrd. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/deploy-cci | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/tests/deploy-cci b/tests/deploy-cci index 8e6acbac4..5e1f5c621 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -35,5 +35,13 @@ deploy() ./install-tiny } +mount_rootfs() +{ + mkdir -p /srv/os /srv/initrd + mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr +} + export_env deploy +mount_rootfs -- 2.23.0

2 2

[PATCH compass-ci] lib/git_bisect: use "--first-parent" option when start bisect
by Cao Xueliang 19 Jan '21

19 Jan '21

1. The alpine 3.13 support to use the latest git which has the "--first-parent" option. 2. This option is useful in avoiding false positives when a merged branch contained broken or non-buildable commits, but the merge itself was OK. Ref: https://git-scm.com/docs/git-bisect Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/delimiter/Dockerfile | 6 +++--- lib/git_bisect.rb | 2 +- 2 files changed, 4 insertions(+), 4 deletions(-) diff --git a/container/delimiter/Dockerfile b/container/delimiter/Dockerfile index ab7519d..cc57320 100644 --- a/container/delimiter/Dockerfile +++ b/container/delimiter/Dockerfile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -FROM alpine:3.11 +FROM alpine:3.13 RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ apk update && \ @@ -9,5 +9,5 @@ RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweiclou 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' RUN umask 002 && \ - echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ - gem install rest-client activesupport git json yaml threadpool elasticsearch faye-websocket terminal-table mail io-console + gem sources -r https://rubygems.org/ -a https://gems.ruby-china.com/ && \ + gem install rest-client activesupport git json yaml elasticsearch faye-websocket terminal-table mail io-console diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index 2fee60c..e577afd 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -82,7 +82,7 @@ class GitBisect puts "bad_commit: #{@bad_commit}" puts "good_commit: #{@good_commit}" - result = `git -C #{@work_dir} bisect start #{@bad_commit} #{@good_commit}` + result = `git -C #{@work_dir} bisect start --first-parent #{@bad_commit} #{@good_commit}` temp = result.split(/\n/) if temp[0].include? 'Bisecting' result = `git -C #{@work_dir} bisect run #{BISECT_RUN_SCRIPT} #{@bad_job_id} #{(a)error_id.inspect} #{@work_dir}` -- 2.23.0

1 0

Re: [PATCH lkp-tests] lib/job.sh: modify the generated file suffix
by Lu Weitao 19 Jan '21

19 Jan '21

On Thu, Jan 14, 2021 at 03:44:10PM +0800, Li Ping wrote: >[why] >user extract $program result actually generating a YAML file. >and compass-ci/sbin/result2stats have already support merge yaml file > >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > lib/job.sh | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > >diff --git a/lib/job.sh b/lib/job.sh >index 03c49e0ad..dc65f68d8 100755 >--- a/lib/job.sh >+++ b/lib/job.sh >@@ -318,5 +318,5 @@ run_test() > run_target_stats() > { > local script_name=$1 >- $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.json >+ $LKP_SRC/stats/$script_name < $TMP_RESULT_ROOT/$script_name > $TMP_RESULT_ROOT/$script_name.yaml it's only write the key-value into $script_name.yaml like: hackbench.throughput: 18277.704125399672 actually we need a matrix with josn file like: { "hackbench.throughput": [ 18277.704125399672 ] } or yaml like: hackbench.throughput: - 18277.704125399672 so we'd better use like: # make sure ENV include RESULT_ROOT # make sure lkp-tests/stats/$script_name exists $LKP_SRC/stats/wrapper $script_name, it will output $RESULT_ROOT/$script_name.json Thanks, Weitao ~ > } >-- >2.23.0 >

2 6

[PATCH v2 compass-ci] doc: sparrow/README.md: update format and content
by Liu Yinsi 18 Jan '21

18 Jan '21

1. git config, this info will be used to build account info, fix verification problem when submit job. 2. setenforce 0, if not set selinux, cci-network.service will not work right. 3. update rootfs download link. 4. division operation instruction Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 66 +++++++++++++++++++++++++++++++++-------------- 1 file changed, 46 insertions(+), 20 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index c7a80e7..89c43d8 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -24,30 +24,49 @@ >openEuler 系统安装详细操作请参考[添加测试用例](https://openeuler.org/zh/docs/20.03_LTS/docs/Installati… ### 操作指导 + +#### 本地搭建compass-ci + 1. 登录 openEuler 系统 -2. 创建工作目录并设置文件权限 +2. 配置git账号 + ```bash + git config --golbal user.name "XXX" + git config --golbal user.email "xxx(a)aa.com" + ``` + +3. 设置文件权限并关闭SELINUX ```bash - mkdir demo && cd demo && umask 002 + umask 002 && setenforce 0 ``` -3. 克隆 compass-ci 项目代码到 demo 目录 + + >**说明：** + >setenforce 0 只是暂时禁用SELINUX，需要修改/etc/selinux/config中的SELINUX=enforcing改为SELINUX=permissive或disabled才能长期有效 + >umask 002 只是暂时设置umask值，需要修改/etc/bashrc中的umask值为002才能长期有效 + +4. 创建工作目录并克隆 compass-ci 项目代码 ```bash + mkdir /c/ && ln -s /c/compass-ci /c/cci git clone https://gitee.com/wu_fengguang/compass-ci.git ``` -4. 执行一键部署脚本 install-tiny + +5. 执行一键部署脚本 install-tiny ```bash cd compass-ci/sparrow && ./install-tiny ``` -5. 生成lkp-aarch64.cgz压缩包 + +#### 提交测试任务前的准备 + +1. 生成lkp-aarch64.cgz压缩包 ```bash cd /c/compass-ci/container/lkp-initrd && ./run ``` -6. 验证账号 +2. 验证账号 ```bash cd /c/compass-ci/sbin && ./build-my-info.rb - 按照提示输入账号名和邮箱即可 + ``` -7. 测试环境是否可以提交job测试 +3. 测试环境是否可以提交job测试 ```bash submit iperf.yaml testbox=vm-2p8g ``` @@ -55,33 +74,40 @@ 执行上述命令正常情况下会提示信息如下: submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci搭建完毕，执行步骤8下载所需要的rootfs文件就可以开始进行测试了。 + compass-ci搭建完毕，执行步骤4下载所需要的rootfs文件就可以开始进行测试了。 -8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） +4. 下载rootfs文件（根据所需要的rootfs在[该目录](http://124.90.34.227:11300/os/)下获取对应版本的cgz文件） ```bash - mkdir -p /srv/os/openeuler/aarch64/ - cd /srv/os/openeuler/aarch64 - wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + mkdir -p /srv/os/openeuler/aarch64/20.03 + cd /srv/os/openeuler/aarch64/20.03 + wget http://124.90.34.227:11300/os/openeuler/aarch64/20.03.cgz ``` -9. 解压cgz包 +5. 解压rootfs cgz 文件 ```bash gzip -dc 20.03.cgz | cpio -idv ``` -10. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… +#### 提交测试任务到本地compass-ci +本文以测试用例iperf.yaml为例 -11. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… +1. [使用 compass-ci 平台测试开源项目](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/… + +2. [编写测试用例](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testca… + +3. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… + ```bash + submit iperf.yaml + ``` -12. 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… - 注意：os_mount 必须指定为cifs +#### 运行测试任务并查看任务结果 -13. 运行测试任务 +1. 运行测试任务 ```bash cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… +2. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) 以查看iperf任务结果为例操如下： ```bash -- 2.23.0

1 0

[PATCH compass-ci] compare_error_messages.rb/error_messages.rb: optimize string catenate
by Lin Jiaxin 18 Jan '21

18 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 2 +- lib/error_messages.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index d9171bf..237b2c9 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -73,7 +73,7 @@ def format_error_messages(error_messages, new_error_ids) new_error_number = 0 error_messages.each do |k, v| - if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + if new_error_ids.include?("build-pkg.#{build_pkg_error_id(k)}") new_error_number += 1 formatted_error_messages = add_sign(formatted_error_messages, '>>', v) else diff --git a/lib/error_messages.rb b/lib/error_messages.rb index d83d890..50f9637 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -44,7 +44,7 @@ class ErrorMessages error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| - if ('build-pkg.' + build_pkg_error_id(k)) == error_id + if "build-pkg.#{build_pkg_error_id(k)}" == error_id error_messages_by_error_id += return_error_line ? [k] : v.to_a end end -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] compare_error_messages.rb/error_messages.rb: optimize string catenate
by Lin Jiaxin 18 Jan '21

18 Jan '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 2 +- lib/error_messages.rb | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index d9171bf..237b2c9 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -73,7 +73,7 @@ def format_error_messages(error_messages, new_error_ids) new_error_number = 0 error_messages.each do |k, v| - if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) + if new_error_ids.include?("build-pkg.#{build_pkg_error_id(k)}") new_error_number += 1 formatted_error_messages = add_sign(formatted_error_messages, '>>', v) else diff --git a/lib/error_messages.rb b/lib/error_messages.rb index d83d890..50f9637 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -44,7 +44,7 @@ class ErrorMessages error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| - if ('build-pkg.' + build_pkg_error_id(k)) == error_id + if "build-pkg.#{build_pkg_error_id(k)}" == error_id error_messages_by_error_id += return_error_line ? [k] : v.to_a end end -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] lib/compare_error_messages.rb: add the credibility check of job
by Lin Jiaxin 18 Jan '21

18 Jan '21

defect: may match the wrong result error_id="build-pkg.pixz.c:warning:'subsuf'defined-but-not-used[-Wunused-function]" filename="pixz.c" correct result: gcc -DHAVE_CONFIG_H -I. -I.. -D_FORTIFY_SOURCE=2 -pthread -Wall -Wno-unknown-pragmas -march=native -O2 -pipe -fstack-protector-strong -fno-plt -Wl,-O1, --sort-common,--as-needed,-z,relro,-z,now -MT pixz-pixz.o -MD -MP -MF .deps/pixz-pixz.Tpo -c -o pixz-pixz.o `test -f 'pixz.c' || echo './'`pixz.c ^^^^^^ wrong result: mv -f .deps/pixz.cpu.Tpo .deps/pixz.cpu.Po ^^^^^^ Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 0b51fc3..d9171bf 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -8,6 +8,40 @@ require_relative 'es_query' require_relative 'error_messages' require_relative "#{ENV['LKP_SRC']}/lib/common" +# previous_job_id's error_ids include error_id +# If later_job_id's output include error filename that extract from error_id, +# the later_job_id is credible. +def credible?(previous_job_id, later_job_id, error_id) + es = ESQuery.new + + previous_result_file = File.join('/srv', es.query_by_id(previous_job_id)['result_root'], 'build-pkg') + later_result_file = File.join('/srv', es.query_by_id(later_job_id)['result_root'], 'build-pkg') + + return false if filenames_check(previous_result_file, later_result_file, error_id).value?(false) + + return true +end + +def filenames_check(previous_result_file, later_result_file, error_id) + filenames = Set.new + filenames_check = Hash.new { |h, k| h[k] = false } + + error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) + error_lines.each do |error_line| + # "src/ssl_sock.c:1454:104: warning: unused parameter ��al�� [-Wunused-parameter]" => "src/ssl_sock.c" + filenames << $1 if error_line =~ /(.*)(:\d+){2}: (error|warning):/ + end + + File.open(later_result_file).each_line do |line| + filenames.each do |filename| + filenames_check[filename] + filenames_check[filename] = true if line.include?(filename) + end + end + + return filenames_check +end + def get_compare_result(previous_job_id, later_job_id) es = ESQuery.new -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] lib/error_messages.rb: obtain_error_messages_by_error_id function add keyword argument
by Lin Jiaxin 18 Jan '21

18 Jan '21

Determine whether to return the error line or error messages by the value of return_error_line Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/error_messages.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/lib/error_messages.rb b/lib/error_messages.rb index 3e1564b..d83d890 100644 --- a/lib/error_messages.rb +++ b/lib/error_messages.rb @@ -40,12 +40,12 @@ class ErrorMessages return @error_messages end - def obtain_error_messages_by_error_id(error_id) + def obtain_error_messages_by_error_id(error_id, return_error_line = false) error_messages_by_error_id = [] error_messages = obtain_error_messages error_messages.each do |k, v| if ('build-pkg.' + build_pkg_error_id(k)) == error_id - error_messages_by_error_id += v.to_a + error_messages_by_error_id += return_error_line ? [k] : v.to_a end end error_messages_by_error_id -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/job.cr: auto install dependency of lkp
by Wei Jihui 18 Jan '21

18 Jan '21

Note that the 'lkp' depends should best be pre-installed in each rootfs. So in theory (if that step is well done) only docker image need this, since we cannot modify 3rd party docker image. However to be robust, we can always include this. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..ae1f170 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -556,7 +556,7 @@ class Job private def get_depends_initrd(program_params, initrd_deps_arr, initrd_pkg_arr) initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - program_params["initramfs-common"] = JSON::Any.new("") if os_mount == "initramfs" + program_params["lkp"] = JSON::Any.new("") program_params.keys.each do |program| if program =~ /^(.*)-\d+$/ -- 2.23.0

1 0

[PATCH lkp-tests] distro/depends: delete initramfs-common
by Wei Jihui 18 Jan '21

18 Jan '21

distro/depends/lkp will pre-installed in each rootfs Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/initramfs-common | 4 ---- 1 file changed, 4 deletions(-) delete mode 100644 distro/depends/initramfs-common diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common deleted file mode 100644 index 5f766da6..00000000 --- a/distro/depends/initramfs-common +++ /dev/null @@ -1,4 +0,0 @@ -time -wget -perf -which -- 2.23.0

1 0

[PATCH compass-ci] doc: sparrow/README.md: update format and content
by Liu Yinsi 18 Jan '21

18 Jan '21

1. git config, this info will be used to build account info, fix verification problem when submit job. 2. setenforce 0, if not set selinux, cci-network.service will not work right. 3. update rootfs download link. 4. division operation instruction Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 68 ++++++++++++++++++++++++++++++++--------------- 1 file changed, 47 insertions(+), 21 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index c7a80e7..f0fea3f 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -24,30 +24,49 @@ >openEuler ��[��](https://openeuler.org/zh/docs/20.03_LTS/docs/Installation/%E5%AE%89%E8%A3%85%E5%87%86%E5%A4%87.html) ### �� + +#### ��compass-ci + 1. �� openEuler �� -2. �� +2. ��git�� ```bash - mkdir demo && cd demo && umask 002 + git config --golbal user.name "XXX" + git config --golbal user.email "xxx(a)aa.com" + ``` + +3. ��SELINUX + ``` + umask 002 && setenforce 0 ``` -3. �� compass-ci �� demo �� + + >**��** + >setenforce 0 ��SELINUX��/etc/selinux/config��SELINUX=enforcing��SELINUX=permissive��disabled�� + >umask 002 ��umask��/etc/bashrc��umask��002�� + +4. �� compass-ci �� ```bash + mkdir /c/ && ln -s /c/compass-ci /c/cci git clone https://gitee.com/wu_fengguang/compass-ci.git ``` -4. �� install-tiny + +5. �� install-tiny ```bash cd compass-ci/sparrow && ./install-tiny ``` -5. ��lkp-aarch64.cgz�� + +#### �� + +1. ��lkp-aarch64.cgz�� ```bash cd /c/compass-ci/container/lkp-initrd && ./run ``` -6. �� +2. �� ```bash cd /c/compass-ci/sbin && ./build-my-info.rb - �� + ``` -7. ��job�� +3. ��job�� ```bash submit iperf.yaml testbox=vm-2p8g ``` @@ -55,33 +74,40 @@ ��: submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci��8��rootfs�� + compass-ci��4��rootfs�� -8. ��rootfs��openeuler/aarch64/20.03��/srv/os/��wget��cgz�� +4. ��rootfs��rootfs��[��](http://124.90.34.227:11300/os/)��cgz�� ```bash - mkdir -p /srv/os/openeuler/aarch64/ - cd /srv/os/openeuler/aarch64 - wget http://124.90.34.227:11300/os/test/openeuler/aarch64/20.03.cgz + mkdir -p /srv/os/openeuler/aarch64/20.03 + cd /srv/os/openeuler/aarch64/20.03 + wget http://124.90.34.227:11300/os/openeuler/aarch64/20.03.cgz ``` -9. ��cgz�� - ```bash +5. ��rootfs cgz �� + ``` gzip -dc 20.03.cgz | cpio -idv ``` -10. [�� compass-ci ��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md) +#### ��compass-ci +��iperf.yaml�� -11. [��](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcase.md) +1. [�� compass-ci ��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-project.zh.md) + +2. [��](https://gitee.com/wu_fengguang/lkp-tests/blob/master/doc/add-testcase.md) + +3. ��[submit��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit-job.zh.md)�� + ``` + submit iperf.yaml + ``` -12. ��[submit��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/submit-job.zh.md)�� - ��os_mount ��cifs +#### �� -13. �� +1. �� ```bash cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) +2. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) ��compass-ci��srv/result��yaml��/��/$testbox/$os-$os-version-$os-arch/$job_id��output��(��tab��) ��iperf�� ```bash -- 2.23.0

1 0

[PATCH lkp-tests 2/4] distro/depends: add vim in initramfs-common
by Zhang Yale 18 Jan '21

18 Jan '21

Signed-off-by: Zhang Yale <ylzhangah(a)qq.com> --- distro/depends/initramfs-common | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/initramfs-common b/distro/depends/initramfs-common index 5f766da6..a2ccf411 100644 --- a/distro/depends/initramfs-common +++ b/distro/depends/initramfs-common @@ -2,3 +2,4 @@ time wget perf which +vim -- 2.23.0

2 3

[PATCH v3 compass-ci] lib/job.cr: enable the account check function
by Wu Zhende 18 Jan '21

18 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 03ca5af..305984c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -392,15 +392,16 @@ class Job private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" + account_info = @es.get_account(self["my_email"]) - # raise account_info unless account_info.is_a?(JSON::Any) + raise error_msg unless account_info.is_a?(JSON::Any) - # @account_info = account_info.as_h - @account_info = account_info.as_h if account_info.is_a?(JSON::Any) + @account_info = account_info.as_h - # raise error_msg if @account_info["found"]? == false - # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - # raise error_msg unless self["my_token"] == @account_info["my_token"] + # my_name can be nil in es + # my_token can't be nil in es + raise error_msg unless self["my_name"] == @account_info["my_name"]?.to_s + raise error_msg unless self["my_token"] == @account_info["my_token"]? @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

1 0

[PATCH compass-ci] container/extract-stats: fix start script
by Ren Wen 18 Jan '21

18 Jan '21

- LAB -> lab - Add 'SRV_HTTP_HOST', 'SRV_HTTP_PORT' to 'names', 'names' is envs what you will get from default configs. Signed-off-by: Ren Wen <15991987063(a)163.com> --- container/extract-stats/start | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/extract-stats/start b/container/extract-stats/start index 6510564..c158977 100755 --- a/container/extract-stats/start +++ b/container/extract-stats/start @@ -14,7 +14,9 @@ names = Set.new %w[ ES_PORT SEND_MAIL_HOST SEND_MAIL_PORT - LAB + SRV_HTTP_HOST + SRV_HTTP_PORT + lab ] defaults = relevant_defaults(names) -- 2.23.0

1 0

[PATCH compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 18 Jan '21

18 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: [ -n "$initrds" ] || exit after: [ -n "$initrds" ] && cat $initrds > initrd || exit [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler: default queue for libvirt
by Ren Wen 18 Jan '21

18 Jan '21

testbox : vt-2p8g -> tbox_group : vt-2p8g -> default queue : vt-2p8g.<arch> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 03ca5af..7d08610 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -310,7 +310,7 @@ class Job # set default value self["queue"] = tbox_group - if tbox_group.to_s.starts_with?(/(vm|dc)-/) + if tbox_group.to_s.starts_with?(/(vm|dc|vt)-/) self["queue"] = "#{tbox_group}.#{arch}" end end -- 2.23.0

1 0

[PATCH compass-ci] scheduler: default queue for libvirt type
by Ren Wen 18 Jan '21

18 Jan '21

testbox : libvirt-2p8g -> default queue : libvirt-2p8g.<arch> Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 1d25236..5667842 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -306,7 +306,7 @@ class Job # set default value self["queue"] = tbox_group - if tbox_group.to_s.starts_with?(/(vm|dc)-/) + if tbox_group.to_s.starts_with?(/(vm|dc|libvirt)-/) self["queue"] = "#{tbox_group}.#{arch}" end end -- 2.23.0

2 2

[PATCH v2 lkp-tests] hosts: add new host file for libvirt
by Xiao Shenwei 18 Jan '21

18 Jan '21

[why] libvirt use a xml file to start vm <os> <kernel>xxx</kernel> <initrd>xxx</initrd> <cmdline>xxx</cmdline> </os> so, we can submit job use os_mount=initramfs to get kernel and initrd absolute path. Minimum memory limit: 16G Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- hosts/vt-2p16g | 5 +++++ hosts/vt-2p32g | 5 +++++ 2 files changed, 10 insertions(+) create mode 100644 hosts/vt-2p16g create mode 100644 hosts/vt-2p32g diff --git a/hosts/vt-2p16g b/hosts/vt-2p16g new file mode 100644 index 000000000..aecbb7948 --- /dev/null +++ b/hosts/vt-2p16g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 16G diff --git a/hosts/vt-2p32g b/hosts/vt-2p32g new file mode 100644 index 000000000..aa7c8c065 --- /dev/null +++ b/hosts/vt-2p32g @@ -0,0 +1,5 @@ +provider: libvirt +template: kvm +nr_node: 1 +nr_cpu: 2 +memory: 32G -- 2.23.0

1 0

[PATCH lkp-tests] jobs: split fault_reproduction.yaml
by Wei Jihui 18 Jan '21

18 Jan '21

[why] user can choose when to ssh, use ssh.yaml is ssh when task normal finish use ssh-on-fail.yaml is ssh when task is failed use ssh-always is ssh always whether task is failed or not. [usage] submit -i ssh.yaml job.yaml Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- jobs/ssh-always.yaml | 4 ++++ jobs/ssh-on-fail.yaml | 12 ++++++++++++ jobs/{fault_reproduction.yaml => ssh.yaml} | 4 ---- 3 files changed, 16 insertions(+), 4 deletions(-) create mode 100644 jobs/ssh-always.yaml create mode 100644 jobs/ssh-on-fail.yaml rename jobs/{fault_reproduction.yaml => ssh.yaml} (80%) diff --git a/jobs/ssh-always.yaml b/jobs/ssh-always.yaml new file mode 100644 index 00000000..0166fe12 --- /dev/null +++ b/jobs/ssh-always.yaml @@ -0,0 +1,4 @@ +# ssh to testbox and sleep always + +<< : jobs/ssh.yaml +<< : jobs/ssh-on-fail.yaml diff --git a/jobs/ssh-on-fail.yaml b/jobs/ssh-on-fail.yaml new file mode 100644 index 00000000..5f28d001 --- /dev/null +++ b/jobs/ssh-on-fail.yaml @@ -0,0 +1,12 @@ +ssh_pub_key: + <%= + begin + File.read("#{ENV['HOME']}/.ssh/id_rsa.pub").chomp + rescue + nil + end + %> + +on_fail: + sshd: + sleep: 6h diff --git a/jobs/fault_reproduction.yaml b/jobs/ssh.yaml similarity index 80% rename from jobs/fault_reproduction.yaml rename to jobs/ssh.yaml index 46a1e042..11e5b175 100644 --- a/jobs/fault_reproduction.yaml +++ b/jobs/ssh.yaml @@ -8,7 +8,3 @@ ssh_pub_key: %> sshd: sleep: 6h - -on_fail: - sshd: - sleep: 6h -- 2.23.0

3 4

[PATCH v2 compass-ci] lib/job.cr: enable the account check function
by Wu Zhende 18 Jan '21

18 Jan '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 03ca5af..db9616b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -392,15 +392,15 @@ class Job private def check_account_info error_msg = "Failed to verify the account.\n" error_msg += "Please refer to https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco…" + account_info = @es.get_account(self["my_email"]) - # raise account_info unless account_info.is_a?(JSON::Any) + raise error_msg unless account_info.is_a?(JSON::Any) - # @account_info = account_info.as_h - @account_info = account_info.as_h if account_info.is_a?(JSON::Any) + @account_info = account_info.as_h - # raise error_msg if @account_info["found"]? == false - # raise error_msg unless self["my_name"] == @account_info["my_name"].to_s - # raise error_msg unless self["my_token"] == @account_info["my_token"] + raise error_msg if @account_info["found"]? == false + raise error_msg if self["my_name"] != @account_info["my_name"].to_s + raise error_msg if self["my_token"] != @account_info["my_token"] @hash.delete("my_uuid") @hash.delete("my_token") -- 2.23.0

2 1

[PATCH compass-ci] lib/job.cr: auto install cgz with lkp
by Wei Jihui 18 Jan '21

18 Jan '21

[why] tests need some common softwares which at lkp-tests/distro/depend/lkp. initramfs-common just for initramfs os, it is not enough. Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- src/lib/job.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 07bfda0..ae1f170 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -556,7 +556,7 @@ class Job private def get_depends_initrd(program_params, initrd_deps_arr, initrd_pkg_arr) initrd_http_prefix = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" mount_type = os_mount == "cifs" ? "nfs" : os_mount.dup - program_params["initramfs-common"] = JSON::Any.new("") if os_mount == "initramfs" + program_params["lkp"] = JSON::Any.new("") program_params.keys.each do |program| if program =~ /^(.*)-\d+$/ -- 2.23.0

2 1