December 2020 - Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 3/3] monitoring/filter.cr: fix send message error
by Wu Zhende 30 Dec '20

30 Dec '20

filter message error: Error writing to socket: Broken pipe Exception: Error writing to socket: Broken pipe (Errno) from /usr/lib/crystal/core/socket.cr:82:13 in 'unbuffered_write' from /usr/lib/crystal/core/io/buffered.cr:192:5 in 'flush' from /usr/lib/crystal/core/io/buffered.cr:200:5 in 'close' from /usr/lib/crystal/core/http/server/handlers/websocket_handler.cr:44:9 in 'call' from /usr/share/crystal/app/lib/kemal/src/kemal/websocket.cr:11:7 in 'call' from /usr/share/crystal/app/lib/kemal/src/kemal/websocket_handler.cr:14:7 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/static_file_handler.cr:64:9 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/exception_handler.cr:8:7 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/log_handler.cr:8:37 in 'call' from /usr/lib/crystal/core/http/server/handler.cr:26:7 in 'call_next' from /usr/share/crystal/app/lib/kemal/src/kemal/init_handler.cr:12:7 in 'call' from /usr/lib/crystal/core/http/server/request_processor.cr:48:11 in 'process' from /usr/lib/crystal/core/http/server/request_processor.cr:22:3 in 'process' from /usr/lib/crystal/core/http/server.cr:497:5 in 'handle_client' from /usr/lib/crystal/core/http/server.cr:463:13 in '->' from /usr/lib/crystal/core/fiber.cr:255:3 in 'run' from /usr/lib/crystal/core/fiber.cr:48:34 in '->' from ??? Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/filter.cr | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/monitoring/filter.cr b/src/monitoring/filter.cr index 5d3bc8b..79af3ab 100644 --- a/src/monitoring/filter.cr +++ b/src/monitoring/filter.cr @@ -41,6 +41,9 @@ class Filter @hash[query].each do |socket| socket.send msg.to_json + rescue e + puts "send msg failed: #{e}" + remove_filter_rule(query, socket) end end -- 2.23.0

1 0

[PATCH compass-ci 2/3] monitoring/monitoring.cr: change rabbitmq queue
by Wu Zhende 30 Dec '20

30 Dec '20

Get logs from different rabbitmq queues. And use multi-process to improve service processing capabilities. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/monitoring/monitoring.cr | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/monitoring/monitoring.cr b/src/monitoring/monitoring.cr index d09c363..8d52d46 100644 --- a/src/monitoring/monitoring.cr +++ b/src/monitoring/monitoring.cr @@ -9,7 +9,8 @@ module Monitoring message_queue_client = MessageQueueClient.new - spawn message_queue_client.monitoring_message_queue(filter, "logging-test", "logging-test") + spawn message_queue_client.monitoring_message_queue(filter, "serial-logging", "serial-logging") + spawn message_queue_client.monitoring_message_queue(filter, "docker-logging", "docker-logging") ws "/filter" do |socket| query = JSON::Any.new("") -- 2.23.0

1 0

[PATCH compass-ci 1/3] master-fluentd/docker-fluentd.conf: send logs to different mq queue
by Wu Zhende 30 Dec '20

30 Dec '20

Send serial logs to serial-logging queue. Send docker logs to docker-logging queue. The two do not affect each other. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 7bf2229..410f2eb 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -18,7 +18,7 @@ <match serial.**> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange serial-logging exchange_type fanout exchange_durable false heartbeat 10 @@ -57,7 +57,7 @@ <store> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange docker-logging exchange_type fanout exchange_durable false heartbeat 10 -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: print ssh info
by Wu Zhende 30 Dec '20

30 Dec '20

After the user logs out, can copy this printed information to log in again. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/monitor.rb b/lib/monitor.rb index 283cd788d..522df713f 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -86,6 +86,7 @@ class Monitor def ssh_connect(ssh_host, ssh_port, web_socket) web_socket.close + puts "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" exec "ssh-keygen -R #{ssh_host}; ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" -- 2.23.0

2 1

[PATCH v2 compass-ci] providers/qemu: encapsulated into functions
by Liu Yinsi 30 Dec '20

30 Dec '20

use function better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/z9.273758 ssssss-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 163 ++++++++++++++++++++++++++---------------- 1 file changed, 100 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..6c4ad00 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,115 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +option_value() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +qemu_options() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) } -[ -n "$initrds" ] || { - exit +run_qemu() +{ + "${kvm[@]}" --append "${append}" } -initrd=initrd -cat $initrds > $initrd +check_logfile +write_logfile -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +qemu_command +option_value +check_option_value -sleep 5 +print_message -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +qemu_options -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

3 4

[PATCH v3 compass-ci] sbin/run_job: Automatic task execution
by Hu Xuejiao 30 Dec '20

30 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 0000000..c6cf9e8 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH compass-ci] container/assist-result: fix failed to start
by Liu Yinsi 30 Dec '20

30 Dec '20

[error] docker: invalid publish opts format (should be name=value but got ':'). [why] $ASSIST_RESULT_PORT not set default value, it's null when deploy compass-ci. [how] set default value ASSIST_RESULT_PORT=8102 in 3-code/dev-env. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..ffaa6e1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${ASSIST_RESULT_PORT:=8102} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -39,6 +40,7 @@ SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] when received email_file's from like "Xing Ming", add has double quotes with it, the email message may end unexpectedly From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready Dear Luan ----email end [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2

[PATCH v2 compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] parsed email address for received email file may like: "Xing Ming <email_address(a)163.com>" or "\"Xing Ming\" <eamil_address(a)163.com>" if the email address contains double quotes as part of string, the email message may end unexpectedly, like: Date: Wed, 23 Dec 2020 18:32:06 +0800^M From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready^M Dear Luan ====email message ends [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2

[PATCH compass-ci 2/4] assign-account: use post request type
by Luan Shengde 30 Dec '20

30 Dec '20

[why] when send requise use rest-client, request data is not supportted for get, so we need to change it to post. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index 4ed95c0..c5d77db 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -12,9 +12,9 @@ require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 -get '/assign_account' do +post '/assign_account' do begin - data = YAML.safe_load request.body.read + data = JSON.parse request.body.read rescue StandardError => e puts e.message puts e.backtrace -- 2.23.0

2 2