- Compass-ci - mailweb.openeuler.org

Re: [PATCH v2 compass-ci] sparrow/4-docker/buildall: container sub-fluentd don't exit
by Li Ping 30 Oct '20

30 Oct '20

dont't ==> doesn't Reply-To: In-Reply-To: <20201030042516.1237314-1-wuzhende666(a)163.com> On Fri, Oct 30, 2020 at 12:25:16PM +0800, Wu Zhende wrote: >[Why] > Now, primary server also need to startup sub-fluentd. > It collects Docker logs and VM serial port logs on the server. > >Signed-off-by: Wu Zhende <wuzhende666(a)163.com> >--- > sparrow/4-docker/buildall | 1 - > 1 file changed, 1 deletion(-) > >diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall >index 6e0a2f8..eb24500 100755 >--- a/sparrow/4-docker/buildall >+++ b/sparrow/4-docker/buildall >@@ -30,7 +30,6 @@ do_one() > mkdir $tmpdir/$container_name 2>/dev/null && > ( > cd $container >- [ "$container_name" == 'sub-fluentd' ] && exit > if [ "$action" != "reboot" ]; then > [ -x build ] && ./build > [ -x install ] && ./install >-- >2.23.0 >

1 0

[PATCH v2 compass-ci] sparrow/4-docker/buildall: container sub-fluentd don't exit
by Wu Zhende 30 Oct '20

30 Oct '20

[Why] Now, primary server also need to startup sub-fluentd. It collects Docker logs and VM serial port logs on the server. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sparrow/4-docker/buildall | 1 - 1 file changed, 1 deletion(-) diff --git a/sparrow/4-docker/buildall b/sparrow/4-docker/buildall index 6e0a2f8..eb24500 100755 --- a/sparrow/4-docker/buildall +++ b/sparrow/4-docker/buildall @@ -30,7 +30,6 @@ do_one() mkdir $tmpdir/$container_name 2>/dev/null && ( cd $container - [ "$container_name" == 'sub-fluentd' ] && exit if [ "$action" != "reboot" ]; then [ -x build ] && ./build [ -x install ] && ./install -- 2.23.0

1 0

[PATCH v4 lkp-tests] hackbench/PKGBUILD: fix source and build function
by Zhang Yale 30 Oct '20

30 Oct '20

[Why] Old source "https://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-1.8.tar.gz" is not available, and need build before package. Signed-off-by: Zhang Yale <zhangyale3(a)huawei.com> --- pkg/hackbench/PKGBUILD | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/pkg/hackbench/PKGBUILD b/pkg/hackbench/PKGBUILD index 797a00e6..94049599 100644 --- a/pkg/hackbench/PKGBUILD +++ b/pkg/hackbench/PKGBUILD @@ -2,13 +2,19 @@ pkgname=hackbench pkgver=1.8 pkgrel=1 arch=('i386' 'x86_64') -url="https://www.kernel.org/pub/linux/utils/rt-tests" +url="https://github.com/lacombar/hackbench.git" license=('GPL') -source=("https://www.kernel.org/pub/linux/utils/rt-tests/rt-tests-${pkgver}.tar.gz") +source=("https://github.com/lacombar/hackbench.git") sha256sums=('SKIP') -package() { +build() +{ + cd "${srcdir}/${pkgname}/${pkgname}" || exit + gcc -o hackbench hackbench.c -lpthread +} + +package() +{ mkdir -p "${pkgdir}/lkp/benchmarks/${pkgname}" - cd "$srcdir/rt-tests-${pkgver}" - make install DESTDIR="${pkgdir}/lkp/benchmarks/${pkgname}" + cp -rf "${srcdir}/${pkgname}/${pkgname}" "${pkgdir}/lkp/benchmarks/${pkgname}" } -- 2.23.0

4 5

[PATCH v3 compass-ci] save hostname and queues relation in redis
by Xiao Shenwei 30 Oct '20

30 Oct '20

[why] the test machine has the ability to know which queues should be requested [how] for VM HW, we will register mac2hostname and hostname2queues. then they can find queues based on mac. API: boot.ipxe/mac/${mac} for docker, we will register hostname2queues, then container can find queues based on hostname. API: boot.container/hostname/${hostname} Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- src/lib/sched.cr | 8 ++++++++ src/scheduler/scheduler.cr | 24 ++++++++++++++++++++++++ 2 files changed, 32 insertions(+) diff --git a/src/lib/sched.cr b/src/lib/sched.cr index b2ef6ce..80cf0b0 100644 --- a/src/lib/sched.cr +++ b/src/lib/sched.cr @@ -38,6 +38,14 @@ class Sched @redis.hash_del("sched/mac2host", normalize_mac(mac)) end + def set_host2queues(hostname : String, queues : String) + @redis.hash_set("sched/host2queues", hostname, queues) + end + + def del_host2queues(hostname : String) + @redis.hash_del("sched/host2queues", hostname) + end + # return: # Hash(String, Hash(String, String)) def get_cluster_state(cluster_id) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index 371e4ed..a946ea0 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -113,6 +113,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/set_host2queues?queues=vm-2p8g.aarch64&host=vm-2p8g.a…" + put "/set_host2queues" do |env| + env.response.headers["Connection"] = "close" + if (client_queues = env.params.query["queues"]?) && (client_host = env.params.query["host"]?) + sched.set_host2queues(client_host, client_queues) + + "Done" + else + "No yet" + end + end + # curl -X PUT "http://localhost:3000/del_host_mac?mac=00-01-02-03-04-05" put "/del_host_mac" do |env| env.response.headers["Connection"] = "close" @@ -125,6 +137,18 @@ module Scheduler end end + # curl -X PUT "http://localhost:3000/del_host2queues?host=vm-2p8g.aarch64" + put "/del_host2queues" do |env| + env.response.headers["Connection"] = "close" + if client_host = env.params.query["host"]? + sched.del_host2queues(client_host) + + "Done" + else + "No yet!" + end + end + # client(runner) report job's status # /~lkp/cgi-bin/lkp-jobfile-append-var # ?job_file=/lkp/scheduled/job.yaml&job_state=running&job_id=10 -- 2.23.0

2 2

[PATCH compass-ci] monitoring/amqp.cr: the retry mechanism is added to rabbitmq
by Wu Zhende 30 Oct '20

30 Oct '20

[Why] If the rabbitmq service is disconnected, monitoring service can't work. When the rabbitmq service restored,need to restart monitoring service. So need to let monitoring service automatic reconnection Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/amqp.cr | 31 +++++++++++++++++++++---------- 1 file changed, 21 insertions(+), 10 deletions(-) diff --git a/src/monitoring/amqp.cr b/src/monitoring/amqp.cr index 1a97dc4..b0341ce 100644 --- a/src/monitoring/amqp.cr +++ b/src/monitoring/amqp.cr @@ -16,18 +16,29 @@ class MessageQueueClient conn.try &.close end + private def filter_msg(conn, filter, exchange_name, queue_name) + conn.channel do |channel| + queue = channel.queue(queue_name) + queue.bind(exchange_name, "") + queue.subscribe(tag: queue_name, block: true) do |msg| + begin + filter.filter_msg(msg.body_io) + rescue e + puts "filter message error: #{e}" + end + end + end + end + def monitoring_message_queue(filter : Filter, exchange_name : String, queue_name : String) - start do |conn| - conn.channel do |channel| - queue = channel.queue(queue_name) - queue.bind(exchange_name, "") - queue.subscribe(tag: queue_name, block: true) do |msg| - begin - filter.filter_msg(msg.body_io) - rescue e - puts "filter message error: #{e}" - end + loop do + begin + start do |conn| + filter_msg(conn, filter, exchange_name, queue_name) end + rescue e + puts "monitoring_message_queue error: #{e}" + sleep 5 end end end -- 2.23.0

1 0

[PATCH lkp-tests] the job-task is waiting when exit_code != 0
by Hu Xuejiao 30 Oct '20

30 Oct '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- bin/run-lkp | 1 + 1 file changed, 1 insertion(+) diff --git a/bin/run-lkp b/bin/run-lkp index 26669331..c4d43db4 100755 --- a/bin/run-lkp +++ b/bin/run-lkp @@ -97,6 +97,7 @@ elif [ "$run_job_exit_code" != 0 ]; then else set_job_state 'failed' fi + $LKP_DEBUG_PREFIX $job_script on_fail elif [ -f '/var/log/wtmp' ] && has_cmd last && last -n1 | grep -q -E '(pts/|tty)[0-9].*(still logged in)'; then set_job_state 'disturbed' disturbed=1 -- 2.23.0

3 2

[PATCH v2 compass-ci] sparrow/0-package: fix failed to config network
by Liu Yinsi 30 Oct '20

30 Oct '20

[why] when execute sparrow/2-network in machine error: ip: command not found [how] openeuler and centos install ip command by iproute, debian install ip command by iproute2 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/0-package/os/centos | 1 + sparrow/0-package/os/debian | 1 + sparrow/0-package/os/openEuler | 1 + 3 files changed, 3 insertions(+) diff --git a/sparrow/0-package/os/centos b/sparrow/0-package/os/centos index 4243f99..6e3ed42 100755 --- a/sparrow/0-package/os/centos +++ b/sparrow/0-package/os/centos @@ -20,6 +20,7 @@ pkgs=( liblockfile lockfile-progs busybox + iproute ) yum install -y --skip-broken "${pkgs[@]}" diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 16ef0a9..8ee6d7e 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -18,6 +18,7 @@ pkgs=( lockfile-progs bridge-utils busybox + iproute2 ) apt-get install -y --fix-missing "${pkgs[@]}" diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler index 79af2b9..43e96ab 100755 --- a/sparrow/0-package/os/openEuler +++ b/sparrow/0-package/os/openEuler @@ -19,6 +19,7 @@ pkgs=( redis wget busybox + iproute ) yum install -y --skip-broken "${pkgs[@]}" -- 2.23.0

1 0

[PATCH lkp-tests] submit on_fail to the scheduler
by Hu Xuejiao 30 Oct '20

30 Oct '20

Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- sbin/submit | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/sbin/submit b/sbin/submit index c49e1e9a..770f0cec 100755 --- a/sbin/submit +++ b/sbin/submit @@ -115,12 +115,14 @@ ARGV.each do |jobfile| sh_run_job = job.sh_run_job sh_extract_stats = job.sh_extract_stats sh_define_files = job.sh_define_files + sh_on_fail = job.sh_on_fail sh_hash = { 'job2sh' => { 'run_job' => sh_run_job, 'extract_stats' => sh_extract_stats, - 'define_files' => sh_define_files + 'define_files' => sh_define_files, + 'on_fail' => sh_on_fail } } -- 2.23.0

3 2

[PATCH lkp-tests] add: add on_fail to job.sh(job.cgz)
by Hu Xuejiao 30 Oct '20

30 Oct '20

[why] if job.yaml define: on_fail: on_fail_sleep: XXXX(time) XXXX: XXXX ... [how] By sh_on_fail, it will appear in job.sh: on_fail() { sleep: on_fail[on_fail_sleep] XXXX: XXXX ... } Signed-off-by: Hu XueJiao <huxuejiao1(a)huawei.com> --- lib/job2sh.rb | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/lib/job2sh.rb b/lib/job2sh.rb index 9c95ff4b..974b78b7 100755 --- a/lib/job2sh.rb +++ b/lib/job2sh.rb @@ -142,7 +142,7 @@ class Job2sh < Job end def parse_one(ancestors, key, val, pass) - if ['pp', 'define_files', 'monitors'].include?(key) + if ['pp', 'define_files', 'monitors', 'on_fail'].include?(key) return false end tabs = indent(ancestors) @@ -266,9 +266,11 @@ class Job2sh < Job sh_run_job(job) local_script_lines += @script_lines - sh_extract_stats(job) local_script_lines += @script_lines + + sh_on_fail() + local_script_lines += @script_lines sh_define_files() @@ -338,6 +340,23 @@ class Job2sh < Job @script_lines end + def sh_on_fail + @script_lines = [] + + job = (@jobx || @job) + on_fail = job['on_fail'] || {} + + out_line 'on_fail()' + out_line '{' + out_line + on_fail.each do |key, val| + if key == "on_fail_sleep" + out_line "\tsleep #{on_fail[key]}" + end + end + out_line '}' + end + def sh_define_files @script_lines = [] -- 2.23.0

4 3

[PATCH compass-ci] auto_submit: it needs specific version number now
by Li Yuanchao 30 Oct '20

30 Oct '20

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index 6d7dc2a..f741b79 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,8 +1,8 @@ AvxToNeon/AvxToNeon: -- testbox=vm-2p8g os=centos os_version=7.6 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml +- testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml compass-ci/compass-ci: -- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- testbox=vm-2p8g os=centos os_version=7.6 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- testbox=vm-2p8g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +- testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +- testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +- testbox=vm-2p8g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml archlinux: -- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 build-pkg.yaml +- testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

3 4