January 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v3 compass-ci] sbin/es-find support range query
by Lu Weitao 19 Jan '21

19 Jan '21

[Example] es-find -r/--range "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 50 ++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) diff --git a/sbin/es-find b/sbin/es-find index 17eefe6..c57c02e 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -5,6 +5,7 @@ # Usage: # es-find key=val ... +# es-find -r "start_time=2021-1-1 00:00:00,2021-1-1 23:59:59;..." # if given multi keys, they create AND conditions # if given val1,val2,... for a key, they create OR conditions # Eg: query (testbox is xxx or yyy) and (suite is aaa) and (user is bbb). @@ -29,7 +30,50 @@ def parse_argv items end +# output: +# eg: +# { +# start_time => { +# gte: '2020-09-10 01:50:00', +# lte: '2020-09-10 01:53:00' +# }, +# ... +# } +def prase_range(range) + range_hash = {} + keyword_list = range.split(';') + keyword_list.each do |keyword| + key, value = keyword.split('=') + check_range_args(key, value) + range_hash[key] = assign_range(value) + end + + range_hash +end + +def check_range_args(key, value) + return if key && value + + warn "error: range value of \"#{key}\" missing", "tips: should give the input of -r like: + \"#{key}=2021-1-1 00:00:00,2021-1-1 23:59:59\" " + exit +end + +def assign_range(value) + range_value = {} + value_list = value.split(',') + if value_list.length > 1 + range_value[:gte] = value_list[0] + range_value[:lte] = value_list[1] + else + range_value[:gte] = value + end + + range_value +end + options = { nr_jobs: 10, desc_keyword: nil } +range = nil opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -49,6 +93,10 @@ opt_parser = OptionParser.new do |opts| options[:desc_keyword] = d end + opts.on('-r', '--range KEYWORD', 'range by keyword: start_time|end_time|...') do |r| + range = r + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -57,6 +105,8 @@ end opt_parser.parse!(ARGV) items = parse_argv +items[:range] = prase_range(range) if range + es = ESQuery.new(ES_HOST, ES_PORT) query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) -- 2.23.0

1 0

Re: [PATCH v3 compass-ci 3/3] compare_error_messages.rb/error_messages.rb: optimize string catenate
by Ren Wen 19 Jan '21

19 Jan '21

style: extra space. Confused subject. Thanks, RenWen Reply-To: In-Reply-To: <20210118114239.3343422-3-ljx.joe(a)qq.com> On Mon, Jan 18, 2021 at 07:42:39PM +0800, Lin Jiaxin wrote: >Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> >--- > lib/compare_error_messages.rb | 2 +- > lib/error_messages.rb | 2 +- > 2 files changed, 2 insertions(+), 2 deletions(-) > >diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb >index d9171bf..237b2c9 100644 >--- a/lib/compare_error_messages.rb >+++ b/lib/compare_error_messages.rb >@@ -73,7 +73,7 @@ def format_error_messages(error_messages, new_error_ids) > new_error_number = 0 > > error_messages.each do |k, v| >- if new_error_ids.include?('build-pkg.' + build_pkg_error_id(k)) >+ if new_error_ids.include?("build-pkg.#{build_pkg_error_id(k)}") > new_error_number += 1 > formatted_error_messages = add_sign(formatted_error_messages, '>>', v) > else >diff --git a/lib/error_messages.rb b/lib/error_messages.rb >index d83d890..50f9637 100644 >--- a/lib/error_messages.rb >+++ b/lib/error_messages.rb >@@ -44,7 +44,7 @@ class ErrorMessages > error_messages_by_error_id = [] > error_messages = obtain_error_messages > error_messages.each do |k, v| >- if ('build-pkg.' + build_pkg_error_id(k)) == error_id >+ if "build-pkg.#{build_pkg_error_id(k)}" == error_id > error_messages_by_error_id += return_error_line ? [k] : v.to_a > end > end >-- >2.23.0 >

1 0

[PATCH v2 lkp-tests] jobs/submit-cci: add job.yaml
by Hu Xuejiao 19 Jan '21

19 Jan '21

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- jobs/submit-cci.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/submit-cci.yaml diff --git a/jobs/submit-cci.yaml b/jobs/submit-cci.yaml new file mode 100644 index 00000000..e45bce72 --- /dev/null +++ b/jobs/submit-cci.yaml @@ -0,0 +1,4 @@ +suite: submit-cci +category: functional + +submit-cci: -- 2.23.0

1 0

[PATCH v2 compass-ci] git_mirror.rb: fix that es-states clean when git-mirror restart
by Li Yuanchao 19 Jan '21

19 Jan '21

Everytime git-mirror get new_refs or just git fetch, these actions and the time will be recorded in hash fork_stat{}. But when container git-mirror restart, fork_stat{} will be initialized. So historical data can't be kept. Now, git-mirror will get historical data from es when initialize hash fork_stat{}. So historical data will be kept. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/git_mirror.rb | 38 +++++++++++++++++++++++++++++--------- 1 file changed, 29 insertions(+), 9 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 86f02e5..1109cfe 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -109,15 +109,8 @@ class MirrorMain @webhook_queue = channel.queue('web_hook') end - def fork_stat_init(stat_key) - @fork_stat[stat_key] = { - queued: false, - priority: 0, - fetch_time: [], - offset_fetch: 0, - new_refs_time: [], - offset_new_refs: 0 - } + def fork_stat_init(git_repo) + @fork_stat[git_repo] = get_fork_stat(git_repo) end def load_defaults(repodir) @@ -372,4 +365,31 @@ class MirrorMain handle_submodule(submodule) end + + def copy_from_es(es_data) + fork_stat = {} + fork_stat[:queued] = es_data['queued'] + fork_stat[:priority] = es_data['priority'] + fork_stat[:fetch_time] = es_data['fetch_time'] + fork_stat[:offset_fetch] = es_data['offset_fetch'] + fork_stat[:new_refs_time] = es_data['new_refs_time'] + fork_stat[:offset_new_refs] = es_data['offset_new_refs'] + return fork_stat + end + + def get_fork_stat(git_repo) + query = { query: { match: { git_repo: git_repo } } } + result = @es_client.search(index: 'repo', body: query)['hits'] + return copy_from_es(result['hits'][0]['_source']) if result['total'].positive? + + fork_stat = { + queued: false, + priority: 0, + fetch_time: [], + offset_fetch: 0, + new_refs_time: [], + offset_new_refs: 0 + } + return fork_stat + end end -- 2.23.0

1 0

[PATCH lkp-tests 2/2] error message enhances
by Zhang Dewan 19 Jan '21

19 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index ab1059a8..3e113093 100755 --- a/distro/debian +++ b/distro/debian @@ -62,7 +62,7 @@ install() # gawk_1%3a4.1.3+dfsg-0.1+b1_amd64.deb echo $(basename $deb_filename) >> opt/deb/keep-deb.$BM_NAME else - echo "error: package is not exist: $deb_pkgname " >&2 + echo "package is not exist: $deb_pkgname " >&2 return 1 fi done < opt/deb/sequence-deb.$BM_NAME -- 2.23.0

1 0

[PATCH lkp-tests 1/2] error message enhances
by Zhang Dewan 19 Jan '21

19 Jan '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- distro/debian | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/debian b/distro/debian index 77cd6258..ab1059a8 100755 --- a/distro/debian +++ b/distro/debian @@ -62,7 +62,7 @@ install() # gawk_1%3a4.1.3+dfsg-0.1+b1_amd64.deb echo $(basename $deb_filename) >> opt/deb/keep-deb.$BM_NAME else - echo "error: $deb_pkgname is not exist." >&2 + echo "error: package is not exist: $deb_pkgname " >&2 return 1 fi done < opt/deb/sequence-deb.$BM_NAME -- 2.23.0

1 0

[PATCH v4 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: firstly, `cat $initrds > $initrd`, secondly, `[ -n "$initrds" ] || exit` after: directly use `[ -n "$initrds" ] && cat $initrds > initrd || exit` Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v2 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: [ -n "$initrds" ] || exit after: [ -n "$initrds" ] && cat $initrds > initrd || exit [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

2 4

[PATCH v3 compass-ci] providers/qemu: fix script hangs when $initrds is null
by Liu Yinsi 19 Jan '21

19 Jan '21

[Error Msg] ++ set_initrd ++ initrd=initrd ++ cat <<< hang here [Root Cause] Function set_initrd() have follow line: cat $initrds > $initrd when $initrds is null, the script will hang when execute cat command. [how] before: firstly, `cat $initrds > $initrd`, secondly, `[ -n "$initrds" ] || exit` after: directly use `[ -n "$initrds" ] && cat $initrds > initrd || exit` [why] 1. before execute cat command, check wether initrds is null. 2. initrd is a brief file name, it can be used directly, so remove $initrd, just us initrd. 3. set_initrd() is unnecessary, so remove. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 854e913..52b10f2 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -60,13 +60,7 @@ check_option_value() # debian has both qemu-system-x86_64 and qemu-system-riscv64 command [[ $kernel =~ 'riscv64' ]] && qemu=qemu-system-riscv64 - [ -n "$initrds" ] || exit -} - -set_initrd() -{ - initrd=initrd - cat $initrds > $initrd + [ -n "$initrds" ] && cat $initrds > initrd || exit } set_bios() @@ -125,7 +119,7 @@ public_option() kvm=( $qemu -kernel $kernel - -initrd $initrd + -initrd initrd -smp $nr_cpu -m $memory -rtc base=localtime @@ -183,7 +177,6 @@ run_qemu() set_options() { - set_initrd set_bios set_helper set_nic -- 2.23.0

1 0

[PATCH v3 compass-ci] sparrow/0-package: add packages
by Liu Yinsi 19 Jan '21

19 Jan '21

[why] start container os-cifs and os-nfs, need to use sudo command, so add it. container/os-cifs/start have follow line: sudo modprobe cifs container/os-nfs/start have follow line: sudo modprobe nfs Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index b25dade..b8ef769 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -24,6 +24,7 @@ pkgs=( hostname wget curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 4d13413..947cbdc 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -24,6 +24,7 @@ pkgs=( qemu-system-aarch64 qemu-system-x86 qemu-system-riscv64 + sudo ) apt-get update diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 04996f5..61a37c4 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -23,6 +23,7 @@ pkgs=( hostname libvirt curl + sudo ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0