- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/3] master-fluentd/docker-fluentd.conf: send logs to different mq queue
by Wu Zhende 30 Dec '20

30 Dec '20

Send serial logs to serial-logging queue. Send docker logs to docker-logging queue. The two do not affect each other. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/master-fluentd/docker-fluentd.conf | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/master-fluentd/docker-fluentd.conf b/container/master-fluentd/docker-fluentd.conf index 7bf2229..410f2eb 100644 --- a/container/master-fluentd/docker-fluentd.conf +++ b/container/master-fluentd/docker-fluentd.conf @@ -18,7 +18,7 @@ <match serial.**> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange serial-logging exchange_type fanout exchange_durable false heartbeat 10 @@ -57,7 +57,7 @@ <store> @type rabbitmq host 172.17.0.1 - exchange logging-test + exchange docker-logging exchange_type fanout exchange_durable false heartbeat 10 -- 2.23.0

1 0

[PATCH lkp-tests] lib/monitor: print ssh info
by Wu Zhende 30 Dec '20

30 Dec '20

After the user logs out, can copy this printed information to log in again. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/monitor.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/lib/monitor.rb b/lib/monitor.rb index 283cd788d..522df713f 100755 --- a/lib/monitor.rb +++ b/lib/monitor.rb @@ -86,6 +86,7 @@ class Monitor def ssh_connect(ssh_host, ssh_port, web_socket) web_socket.close + puts "ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" exec "ssh-keygen -R #{ssh_host}; ssh root@#{ssh_host} -p #{ssh_port} -o StrictHostKeyChecking=no" -- 2.23.0

2 1

[PATCH v2 compass-ci] providers/qemu: encapsulated into functions
by Liu Yinsi 30 Dec '20

30 Dec '20

use function better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/z9.273758 ssssss-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 163 ++++++++++++++++++++++++++---------------- 1 file changed, 100 insertions(+), 63 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fd921dc..6c4ad00 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,78 +7,115 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi +} -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm +} -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +option_value() +{ + append= + initrds= + while read a b c + do + case "$a" in + '#') + ;; + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + *) + ;; + esac + done < $ipxe_script +} -append= -initrds= -while read a b c -do - case "$a" in - '#') - ;; - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - *) - ;; - esac -done < $ipxe_script +check_option_value() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +qemu_options() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) } -[ -n "$initrds" ] || { - exit +run_qemu() +{ + "${kvm[@]}" --append "${append}" } -initrd=initrd -cat $initrds > $initrd +check_logfile +write_logfile -echo $SCHED_PORT -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file +qemu_command +option_value +check_option_value -sleep 5 +print_message -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) +qemu_options -"${kvm[@]}" --append "${append}" +run_qemu -- 2.23.0

3 4

[PATCH v3 compass-ci] sbin/run_job: Automatic task execution
by Hu Xuejiao 30 Dec '20

30 Dec '20

[purpose] Verify that a local job can be consumed properly Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sbin/run_job | 73 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 73 insertions(+) create mode 100755 sbin/run_job diff --git a/sbin/run_job b/sbin/run_job new file mode 100755 index 0000000..c6cf9e8 --- /dev/null +++ b/sbin/run_job @@ -0,0 +1,73 @@ +#!/bin/bash + +cci_makepkg_file="$LKP_SRC/jobs/cci-makepkg.yaml" +grep "atomic" $cci_makepkg_file || sed -i 's|benchmark:|& atomic|g' $cci_makepkg_file + +submit_one_job() +{ + cmd=( + submit -m $yaml + os=${single_arg[0]} + os_arch=${single_arg[1]} + os_version=${single_arg[2]} + os_mount=${single_arg[3]} + testbox=$testbox + queue="${tbox_group}.${HOSTNAME}" + ) + output=$("${cmd[@]}") + echo "$output" | grep -o -E 'job_state=finished[^\"]*' +} + +submit_job() +{ + yaml_args=("$cci_makepkg_file" "host-info.yaml") + for yaml in "${yaml_args[@]}" + do + submit_one_job & + done +} + +run_qemu() +{ + local nr_vm=4 + local testbox="vm-2p8g" + local tbox_group="vm-2p8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-qemu -n "$hostname" -c "$nr_vm" -q "$queues" + + qemu_args=( + "openeuler aarch64 20.03 cifs" + "centos aarch64 7.6.1810 initramfs" + ) + for qemu_arg in "${qemu_args[@]}" + do + local single_arg=($(echo $qemu_arg)) + submit_job + done +} + +run_docker() +{ + local nr_container=2 + local tbox_group="dc-8g" + local testbox="dc-8g" + local hostname="${tbox_group}.${HOSTNAME}" + local queues="${tbox_group}.${HOSTNAME}" + + $CCI_SRC/providers/multi-docker -n $hostname -c $nr_container -q $queues + + docker_args=( + "centos aarch64 7.6.1810 container" + ) + for docker_arg in "${docker_args[@]}" + do + local single_arg=($(echo $docker_arg)) + submit_job + done +} + +run_qemu +run_docker +wait -- 2.23.0

2 1

[PATCH compass-ci] container/assist-result: fix failed to start
by Liu Yinsi 30 Dec '20

30 Dec '20

[error] docker: invalid publish opts format (should be name=value but got ':'). [why] $ASSIST_RESULT_PORT not set default value, it's null when deploy compass-ci. [how] set default value ASSIST_RESULT_PORT=8102 in 3-code/dev-env. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 750738f..ffaa6e1 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -18,6 +18,7 @@ server_ip=$(ip route get 1.2.3.4 | awk '{print $7; exit}') : ${SEND_MAIL_PORT:=49000} : ${LOCAL_SEND_MAIL_PORT:=11311} : ${LOCAL_ROBOT_EMAIL_ADDRESS:=Crystal TEAM} +: ${ASSIST_RESULT_PORT:=8102} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF @@ -39,6 +40,7 @@ SEND_MAIL_PORT: $SEND_MAIL_PORT LOCAL_SEND_MAIL_PORT: $LOCAL_SEND_MAIL_PORT LOCAL_ROBOT_EMAIL_ADDRESS: $LOCAL_ROBOT_EMAIL_ADDRESS MAILDIR: /srv/cci/Maildir/.compass-ci +ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT EOF mkdir -p /etc/compass-ci/account -- 2.23.0

1 0

[PATCH compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] when received email_file's from like "Xing Ming", add has double quotes with it, the email message may end unexpectedly From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready Dear Luan ----email end [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2

[PATCH v2 compass-ci] mail-robot: fix bug
by Luan Shengde 30 Dec '20

30 Dec '20

[why] parsed email address for received email file may like: "Xing Ming <email_address(a)163.com>" or "\"Xing Ming\" <eamil_address(a)163.com>" if the email address contains double quotes as part of string, the email message may end unexpectedly, like: Date: Wed, 23 Dec 2020 18:32:06 +0800^M From: compass-ci-reply(a)qq.com To: shdluan(a)163.com Subject: [compass-ci] Account Ready^M Dear Luan ====email message ends [how] delete extra double quotes Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/parse-apply-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 8c55d8f..7024731 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -36,7 +36,7 @@ class ParseApplyAccountEmail @my_info = { 'my_email' => mail_content.from[0], - 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, ''), + 'my_name' => mail_content.From.unparsed_value.gsub(/ <[^<>]*>/, '').gsub(/"/, ''), 'my_ssh_pubkey' => [] } end -- 2.23.0

2 2

[PATCH compass-ci 2/4] assign-account: use post request type
by Luan Shengde 30 Dec '20

30 Dec '20

[why] when send requise use rest-client, request data is not supportted for get, so we need to change it to post. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index 4ed95c0..c5d77db 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -12,9 +12,9 @@ require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 -get '/assign_account' do +post '/assign_account' do begin - data = YAML.safe_load request.body.read + data = JSON.parse request.body.read rescue StandardError => e puts e.message puts e.backtrace -- 2.23.0

2 2

[PATCH v4 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 30 Dec '20

30 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..801649cf4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || return + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

2 2

[PATCH lkp-tests] makepkg.conf: use the my_curl command to use the cache
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- etc/makepkg.conf | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 1fffdc2d2..5d73399d9 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -8,10 +8,10 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/curl -gqC - -o %o %u' - 'ftp::/usr/bin/curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' +DLAGENTS=('file::/usr/bin/my_curl -gqC - -o %o %u' + 'ftp::/usr/bin/my_curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' 'rsync::/usr/bin/rsync --no-motd -z %u %o' 'scp::/usr/bin/scp -C %u %o') -- 2.23.0

2 1

[PATCH compass-ci] delimiter/utils: set job testbox
by Cao Xueliang 29 Dec '20

29 Dec '20

If job has config field, set the testbox to "vm-2p32g", otherwise set the testbox to "vm-2p16g". Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 17128ab..49d48ee 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -135,12 +135,14 @@ module Utils job['my_email'] = account_info['my_email'] job['my_token'] = account_info['my_token'] job['bad_job_id'] = job_id + job['testbox'] = job['config'] ? 'vm-2p32g' : 'vm-2p16g' job.delete('error_ids') job.delete('start_time') job.delete('job_state') job.delete('end_time') job.delete('loadavg') + job.delete('queue') job.delete('stats') job.delete('id') -- 2.23.0

1 0

[PATCH lkp-tests] lib/bootstrap.sh: try to install ntpdate if there is no such pkg
by Yu Chuan 29 Dec '20

29 Dec '20

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- lib/bootstrap.sh | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index ae90220750db..7d15bb370fa9 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -6,6 +6,7 @@ . $LKP_SRC/lib/reboot.sh . $LKP_SRC/lib/ucode.sh . $LKP_SRC/lib/tbox.sh +. $LKP_SRC/lib/detect-system.sh # borrowed from linux/tools/testing/selftests/rcutorture/doc/initrd.txt # Author: Paul E. McKenney <paulmck(a)linux.vnet.ibm.com> @@ -236,7 +237,13 @@ run_ntpdate() [ -n "$LKP_SERVER" ] || return is_clearlinux && clearlinux_timesync && return - [ -x '/usr/sbin/ntpdate' ] || return + has_cmd ntpdate || { + detect_system + local DISTRO=$_system_name_lowercase + . $LKP_SRC/distro/installer/$DISTRO $(echo ntpdate | $LKP_SRC/sbin/adapt-packages $DISTRO) + } + + has_cmd ntpdate || return local hour="$(date +%H)" -- 2.23.0

2 2

[PATCH v2 lkp-tests 2/2] my_curl: change the curl request so that the cache can be used
by Cui Yili 29 Dec '20

29 Dec '20

change curl request, in order to use local cache. e.g. https://download.redis.io/releases/redis-4.0.2.tar.gz ==> https://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k http://download.redis.io/releases/redis-4.0.2.tar.gz ==> http://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k Signed-off-by: cuiyili <2268260388(a)qq.com> --- rootfs/addon/usr/bin/my_curl | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 rootfs/addon/usr/bin/my_curl diff --git a/rootfs/addon/usr/bin/my_curl b/rootfs/addon/usr/bin/my_curl new file mode 100755 index 000000000..97d5352a4 --- /dev/null +++ b/rootfs/addon/usr/bin/my_curl @@ -0,0 +1,22 @@ +#!/bin/bash + +declare -a array +i=0 + +if [ -n "$HTTP_PROXY_HOST" ]; then + for param in "$@"; + do + if [[ "$param" =~ http://(.*) ]] && [ -n "$HTTP_PROXY_PORT" ]; then + array[$i]="http://$HTTP_PROXY_HOST:$HTTP_PROXY_PORT/${BASH_REMATCH[1]}" + elif [[ "$param" =~ https://(.*) ]] && [ -n "$HTTPS_PROXY_PORT" ]; then + array[$i]="https://$HTTP_PROXY_HOST:$HTTPS_PROXY_PORT/${BASH_REMATCH[1]}" + else + array[$i]="$param" + fi + ((i++)) + done + + command curl -k ${array[*]} +else + command curl "$@" +fi -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/2] lkp-setup-rootfs: all use of env http_proxy and delete specific proxy settings
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- bin/lkp-setup-rootfs | 9 --------- 1 file changed, 9 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..a7f1a7ffb 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -105,7 +105,6 @@ install_proxy_yum() { [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo fi @@ -117,11 +116,6 @@ install_proxy_apt() { [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy -Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; -Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; -EOF - for cfg in /etc/apt/sources.list /etc/apt/sources.list.d/* do [ -f "$cfg" ] || continue @@ -176,9 +170,6 @@ install_proxy() install_goproxy - [ -z "$SQUID_HOST" ] && return - [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum } -- 2.23.0

1 0

[PATCH compass-ci] delimiter/utils: add query job stats multiple times
by Cao Xueliang 29 Dec '20

29 Dec '20

Sometimes, the job state is already extract_finished, but we can't query the job stats, so add a temporary solutions, query the job stats multiple times. The extract container will be improved in future. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 372e821..17128ab 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -83,10 +83,10 @@ module Utils extract_finished = monitor_run_stop(query) return nil unless extract_finished.zero? - es = ESQuery.new - new_job = es.query_by_id(new_job_id) + stats = query_stats(new_job_id, 10) + raise "es cant query #{new_job_id} stats field!" unless stats - status = new_job['stats'].key?(error_id) ? 'bad' : 'good' + status = stats.key?(error_id) ? 'bad' : 'good' puts "new_job_id: #{new_job_id}" puts "upstream_commit: #{job['upstream_commit']}" record_jobs(new_job_id, job['upstream_commit']) @@ -94,6 +94,21 @@ module Utils return status end + # sometimes the job_state is extract_finished + # but we cant query the job stats field in es, so, add many times query + # this is a temporary solution, the extract container will be improved in future. + def query_stats(job_id, times) + (1..times).each do |i| + new_job = ESQuery.new.query_by_id(job_id) + puts "query stats times: #{i}" + return new_job['stats'] if new_job['stats'] + + sleep 60 + end + + return nil + end + def record_jobs(job_id, job_commit) FileUtils.mkdir_p TMP_RESULT_ROOT unless File.exist? TMP_RESULT_ROOT commit_jobs = File.join(TMP_RESULT_ROOT, 'commit_jobs') -- 2.23.0

1 0

[PATCH lkp-tests 2/2] my_curl: change the curl request so that the cache can be used.
by Cui Yili 29 Dec '20

29 Dec '20

change curl reuqest, in order to use local cache. E..g https://download.redis.io/releases/redis-4.0.2.tar.gz ==> https://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k http://download.redis.io/releases/redis-4.0.2.tar.gz ==> http://ip:port/download.redis.io/releases/redis-4.0.2.tar.gz -k Signed-off-by: cuiyili <2268260388(a)qq.com> --- rootfs/addon/usr/bin/my_curl | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100755 rootfs/addon/usr/bin/my_curl diff --git a/rootfs/addon/usr/bin/my_curl b/rootfs/addon/usr/bin/my_curl new file mode 100755 index 000000000..697f53d65 --- /dev/null +++ b/rootfs/addon/usr/bin/my_curl @@ -0,0 +1,22 @@ +#!/bin/bash + +declare -a array +j=0 + +if [ -n "$HTTP_PROXY_HOST" ]; then + for param in "$@"; + do + if [[ "$param" =~ http://(.*) ]] && [ -n "$HTTP_PROXY_PORT" ]; then + array[$j]="http://$HTTP_PROXY_HOST:$HTTP_PROXY_PORT/${BASH_REMATCH[1]}" + elif [[ "$param" =~ https://(.*) ]] && [ -n "$HTTPS_PROXY_PORT" ]; then + array[$j]="https://$HTTP_PROXY_HOST:$HTTPS_PROXY_PORT/${BASH_REMATCH[1]}" + else + array[$j]="$param" + fi + ((j++)) + done + + command curl -k ${array[*]} +else + command curl "$@" +fi -- 2.23.0

2 1

[PATCH lkp-tests 1/2] lkp-setup-rootfs: all use of env http_proxy and delete specific proxy settings
by Cui Yili 29 Dec '20

29 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- bin/lkp-setup-rootfs | 9 --------- 1 file changed, 9 deletions(-) diff --git a/bin/lkp-setup-rootfs b/bin/lkp-setup-rootfs index cca1b53be..a7f1a7ffb 100755 --- a/bin/lkp-setup-rootfs +++ b/bin/lkp-setup-rootfs @@ -105,7 +105,6 @@ install_proxy_yum() { [ -d "/etc/yum/" ] || return - echo "proxy=http://$SQUID_HOST:$SQUID_PORT" >> /etc/yum.conf if ls /etc/yum.repos.d/*.repo >/dev/null 2>&1; then sed -i 's/https:/http:/g' /etc/yum.repos.d/*.repo fi @@ -117,11 +116,6 @@ install_proxy_apt() { [ -d "/etc/apt/" ] || return - cat << EOF > /etc/apt/apt.conf.d/proxy -Acquire::http::proxy "http://$SQUID_HOST:$SQUID_PORT/"; -Acquire::ftp::proxy "ftp://$SQUID_HOST:$SQUID_PORT/"; -EOF - for cfg in /etc/apt/sources.list /etc/apt/sources.list.d/* do [ -f "$cfg" ] || continue @@ -176,9 +170,6 @@ install_proxy() install_goproxy - [ -z "$SQUID_HOST" ] && return - [ -z "$SQUID_PORT" ] && return - install_proxy_apt || install_proxy_yum } -- 2.23.0

1 0

[PATCH compass-ci] scheduler: fix: don't return queues individually
by Ren Wen 29 Dec '20

29 Dec '20

- avoid to auto submit idle job if finding no job from sub-queues, because idle-queues are not found yet. - sub-queues should be shuffled. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 21396c4..a1e3497 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -72,14 +72,14 @@ class Sched end end - return sub_queues unless sub_queues.empty? - idle_queues = [] of String default_queues.each do |queue| idle_queues << "#{queue}/idle" end - return idle_queues + all_queues = rand_queues(sub_queues) + idle_queues + + return all_queues end def get_job_from_queues(queues, testbox) -- 2.23.0

1 0

[PATCH compass-ci 2/2] scheduler: fix confused comments and add comments
by Ren Wen 29 Dec '20

29 Dec '20

fix: - use 'is joined with' instead of 'is link with'. - fix format. add: - add a TODO comment. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index f60c0e5..a7b4bd8 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -25,10 +25,11 @@ class Sched @log.warn(e) end - # auto submit a job to collect the host information - # grub hostname is link with ":", like "00:01:02:03:04:05" - # remind: if like with "-", last "-05" is treated as host number - # then hostname will be "sut-00-01-02-03-04" !!! + # auto submit a job to collect the host information. + # + # grub hostname is joined with ":", like "00:01:02:03:04:05". + # remind: if joined with "-", last "-05" is treated as host number + # then hostname will be "sut-00-01-02-03-04" !!! def submit_host_info_job(mac) host = "sut-#{mac}" @redis.hash_set("sched/mac2host", normalize_mac(mac), host) @@ -63,6 +64,7 @@ class Sched sub_queues = [] of String default_queues.each do |queue| + # TODO: this could be high cost and should be improved in future : keys(pattern). matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") next if matched_queues.empty? -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler: find_job_boot: remove useless functions
by Ren Wen 29 Dec '20

29 Dec '20

- find_job : used to find job according to testbox. Now use sub-queues. - get_idle_job : used to find and submit idle jobs. Now use idle-queues. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 38 ---------------------------------- 1 file changed, 38 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 21396c4..f60c0e5 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -183,32 +183,6 @@ class Sched end end - private def find_job(testbox : String, count = 1) - tbox_group = JobHelper.match_tbox_group(testbox) - tbox = tbox_group.partition("--")[0] - - queue_list = query_consumable_keys(tbox) - - boxes = ["sched/" + testbox, - "sched/" + tbox_group, - "sched/" + tbox, - "sched/" + tbox_group + "/idle"] - boxes.each do |box| - next if queue_list.select(box).size == 0 - count.times do - job = prepare_job(box, testbox) - return job if job - - sleep(1) unless count == 1 - end - end - - # when find no job, auto submit idle job at background - spawn { auto_submit_idle_job(tbox_group) } - - return nil - end - private def prepare_job(queue_name, testbox) response = @task_queue.consume_task(queue_name) job_id = JSON.parse(response[1].to_json)["id"] if response[0] == 200 @@ -232,18 +206,6 @@ class Sched return job end - private def get_idle_job(tbox_group, testbox) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - - # if there has no idle job, auto submit and get 1 - if job.nil? - auto_submit_idle_job(tbox_group) - job = prepare_job("sched/#{tbox_group}/idle", testbox) - end - - return job - end - private def auto_submit_idle_job(testbox) full_path_patterns = "#{CCI_REPOS}/#{LAB_REPO}/allot/idle/#{testbox}/*.yaml" fields = ["testbox=#{testbox}", "subqueue=idle"] -- 2.23.0

1 0

[PATCH v3 compass-ci] lib/params_group.rb optimize filter_groups()
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] 1. We had for-each key-value in hash, needn't query value by key once more. 2. When remove empty value of Hash(String, Hash(String, Array(job)), remove the first level key-value at first. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..9be2cbe 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,11 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + if value.empty? + groups.delete(group_key) + next end - groups.delete(group_key) if groups[group_key].empty? + value.delete_if { |_dim_key, job_list| job_list.empty? } end end -- 2.23.0

1 0

[PATCH compass-ci] sbin/auto_submit: take the pkgbuild_repo if it is linux
by Lin Jiaxin 29 Dec '20

29 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index f96be8c..51ecc6a 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -20,7 +20,7 @@ class AutoSubmit def get_pkgbuild_repo(repo_array) pkgbuild_repo = "archlinux/#{repo_array[0]}" repo_array.each do |repo| - next unless repo =~ /-git$/ + next unless repo =~ /(-git|linux)$/ pkgbuild_repo = "archlinux/#{repo}" break -- 2.23.0

2 1

[PATCH lkp-tests] auto_submit: add build-linux.yaml for linux
by Lin Jiaxin 29 Dec '20

29 Dec '20

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- jobs/build-linux.yaml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 jobs/build-linux.yaml diff --git a/jobs/build-linux.yaml b/jobs/build-linux.yaml new file mode 100644 index 00000000..080a84b3 --- /dev/null +++ b/jobs/build-linux.yaml @@ -0,0 +1,4 @@ +suite: build-pkg +category: functional +build-pkg: +config: <%= 'aarch64-randconfig-'+SecureRandom.random_number(1000).to_s %> -- 2.23.0

2 1

[PATCH compass-ci] lib/es_query.rb: get multi-field aggregation count result
by Lu Kaiyi 29 Dec '20

29 Dec '20

[why] get multi-field aggregation count for overview of report page. [example] es = ESQuery.new query_items = { 'os' => 'openeuler' } field1 = 'suite' field2 = 'job_state' es.query_fields(field1, field2, query_items) [input]: field1, field2, query_items (optional for query_items, default no scope limitation) [output]: {"build-pkg"=> {"failed"=>3804, "finished"=>800, "incomplete"=>196, "submit"=>136, "OOM"=>11}, "cci-depends"=>{"finished"=>580, "failed"=>218, "incomplete"=>1}, "iperf"=>{"finished"=>136, "failed"=>57}, "spinlock"=>{"finished"=>52, "failed"=>1}, ... } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- lib/es_query.rb | 59 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 59 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index 88a420f..d881f6b 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -87,6 +87,52 @@ class ESQuery result end + + # select field1, field2, count(*) from index where query_items group by field1, field2 + # example: + # es = ESQuery.new + # query_items = { + # 'os' => 'openeuler' + # } + # field1 = 'suite' + # field2 = 'job_state' + # es.query_fields(field1, field2, query_items) + # input: + # field1, field2, query_items + # (optional for query_items, default no scope limitation) + # output: + # {"build-pkg"=> + # {"failed"=>3804, "finished"=>800, "incomplete"=>196, "submit"=>136, "OOM"=>11}, + # "cci-depends"=>{"finished"=>580, "failed"=>218, "incomplete"=>1}, + # "iperf"=>{"finished"=>136, "failed"=>57}, + # "spinlock"=>{"finished"=>52, "failed"=>1}, + # ... + # } + def query_fields(field1, field2, query_items = {}) + query = { + query: { + bool: { + must: build_multi_field_subquery_body(query_items) + } + }, + aggs: { + "all_#{field1}" => { + terms: { field: field1, size: 1000 }, + aggs: { + "all_#{field2}" => { + terms: { field: field2, size: 1000 } + } + } + } + }, + size: 0 + } + es_result = @client.search(index: @index + '*', body: query)['aggregations']["all_#{field1}"]['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse! + + parse_fields(es_result, field2) + end end # Range Query Example: @@ -146,3 +192,16 @@ def assign_desc_body(keyword) }] } end + +def parse_fields(es_result, field) + result_hash = {} + es_result.each do |result| + key = result['key'] + all_field = result["all_#{field}"]['buckets'] + field_count = {} + all_field.map { |item| field_count[item['key']] = item['doc_count'] } + result_hash[key] = field_count unless field_count.empty? + end + + result_hash +end -- 2.23.0

2 1

[PATCH compass-ci] scheduler: fix: only use nodes job specified
by Ren Wen 29 Dec '20

29 Dec '20

[Before]: Every node in cluster config file will be distributed a job when submitting cluster jobs. [Now]: Job will be distributed according to job content. For example: If job has: if role server: if role client: then only nodes which roles contains 'server' and 'client' will use. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c09d145..bcb265f 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -38,6 +38,7 @@ class Sched job_messages = Array(Hash(String, String)).new lab = job.lab subqueue = job.subqueue + roles = get_roles(job) # collect all job ids job_ids = [] of String @@ -52,6 +53,9 @@ class Sched # steps for each host cluster_config["nodes"].as_h.each do |host, config| + # continue if role in cluster config matches role in job + next if (config["roles"].as_a.map(&.to_s) & roles).empty? + queue = host.to_s job_id = add_task("#{queue}/#{subqueue}", lab) @@ -157,6 +161,11 @@ class Sched end end + def get_roles(job) + roles = job.hash.keys.map { |key| $1 if key =~ /^if role (.*)/ } + roles.compact.map(&.strip) + end + def get_commit_date(job) if (job["upstream_repo"] != "") && (job["upstream_commit"] != "") data = JSON.parse(%({"git_repo": "#{job["upstream_repo"]}.git", -- 2.23.0

1 0

[PATCH lkp-tests] optimize the deploy-cci script
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- tests/deploy-cci | 53 ++++++++++++++++++++++++++++-------------------- 1 file changed, 31 insertions(+), 22 deletions(-) diff --git a/tests/deploy-cci b/tests/deploy-cci index 55835544..21308ecf 100755 --- a/tests/deploy-cci +++ b/tests/deploy-cci @@ -2,30 +2,39 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -# redirect remote-repository address to "/srv/git/gitee.com" -if [ -n "$GIT_SERVER" ]; then - cat >> /etc/gitconfig <<-EOF - [url "git://$GIT_SERVER/gitee.com"] - insteadOf=https://gitee.com - EOF -fi +env() +{ + export initrd_http_host=$INITRD_HTTP_HOST + export initrd_http_port=$INITRD_HTTP_PORT + export gitcache_host=$GITCACHE_HOST + export gitcache_port=$GITCACHE_PORT +} -export initrd_http_host=$INITRD_HTTP_HOST -export initrd_http_port=$INITRD_HTTP_PORT -export gitcache_host=$GITCACHE_HOST -export gitcache_port=$GITCACHE_PORT +deploy() +{ + # redirect remote-repository address to "/srv/git/gitee.com" + if [ -n "$GIT_SERVER" ]; then + cat >> /etc/gitconfig <<-EOF + [url "git://$GIT_SERVER/gitee.com"] + insteadOf=https://gitee.com + EOF + fi -mkdir -p /c -cd /c || exit -git clone https://gitee.com/wu_fengguang/compass-ci.git + mkdir -p /c + cd /c || exit + git clone https://gitee.com/wu_fengguang/compass-ci.git -cd /c/compass-ci || exit -[ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + cd /c/compass-ci || exit + [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" -# Need mount tmpfs to fix compass-ci/container/os-nfs/start error -# "exportfs: /exports/result does not support NFS export" -mount -t tmpfs none /srv + # Need mount tmpfs to fix compass-ci/container/os-nfs/start error + # "exportfs: /exports/result does not support NFS export" + mount -t tmpfs none /srv -# test the script for one-click deployment -cd /c/compass-ci/sparrow || exit -./install-tiny + # test the script for one-click deployment + cd /c/compass-ci/sparrow || exit + ./install-tiny +} + +env +deploy -- 2.23.0

2 1

[PATCH v1 compass-ci 2/2] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 8 ++++++-- 2 files changed, 13 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..a4a604b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -43,6 +45,8 @@ class MailBisectResult all errors/warnings: #{@all_errors} + reference information: + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 1

[PATCH compass-ci] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 7 +++++-- 2 files changed, 12 insertions(+), 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..2d2e855 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -43,6 +45,7 @@ class MailBisectResult all errors/warnings: #{@all_errors} + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 1

[PATCH compass-ci] container: mount "/srv/git/archlinux" to srv-http
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 8c8ca0b..d9f461b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -15,6 +15,7 @@ cmd=( -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro + -v /srv/git/archlinux:/srv/git/archlinux:ro -d srv-http ) -- 2.23.0

1 1

[PATCH v3 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..8edd28e 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,11 @@ dirs=( ) mkdir -p "${dirs[@]}" + +if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd +elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t "$os_mount" //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr +fi -- 2.23.0

2 1

[PATCH v2 compass-ci 2/2] mail-robot: optimize parse email_file
by Luan Shengde 29 Dec '20

29 Dec '20

[why] avoid build too much messages in the code files [how] just raise error code in the code file, and build the error message in the file building email message Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 4 +-- .../lib/parse-apply-account-email.rb | 31 ++++++------------- 2 files changed, 11 insertions(+), 24 deletions(-) diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb index 7cac0d5..af31e17 100755 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -38,9 +38,7 @@ class GiteeCommitUrlCheck def gitee_commit_exist(email_index) return if email_index - error_message = "We can not confirm whether the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with your email.' + error_message = 'COMMIT_URL_NOT_AVAILABLE' raise error_message end diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 7024731..a41a89a 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey @@ -27,7 +28,7 @@ require_relative 'gitee-commit-url-check' # gitee.com: # call GiteeCommitUrlCheck to check the commit # non-gitee.com: -# execute curl to check the commit +# get request date for the commit url to check the commit # parse_pub_key # check my_ssh_pubkey exists and return it class ParseApplyAccountEmail @@ -63,10 +64,8 @@ class ParseApplyAccountEmail # the commit url should be in a standart format, example: # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - error_message = "No matched commit url found.\n" - error_message += 'Ensure that you have add a right commit url, ' - error_message += "and with prefix 'my oss commit:'." - raise error_message + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) + raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) end mail_content_line.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] @@ -88,16 +87,13 @@ class ParseApplyAccountEmail return unless match_out.empty? - error_message = "The repo url for your commit is not in our upstream-repo list.\n" - error_message += 'Use a new one, or consulting the manager for available repo list.' - - raise error_message + raise 'NOT_REGISTERED' end def commit_url_availability(url, base_url) hub_name = url.split('/')[2] - # it requires authentication when execute curl to get the commit information + # it requires authentication when sending request to get the commit information # clone the repo and then validate the commit for the email address if hub_name.eql? 'gitee.com' gitee_commit(url, base_url) @@ -112,24 +108,17 @@ class ParseApplyAccountEmail end def non_gitee_commit(url) - url_fdback = %x(curl #{url}) + url_fdback = RestClient.get(url).body email_index = url_fdback.index @my_info['my_email'] return unless email_index.nil? - error_message = "We can not confirm the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with you email.' - - raise error_message + raise 'COMMIT_URL_NOT_AVAILABLE' end def parse_pub_key - error_message = "No pub_key found.\n" - error_message += 'Please add a pub_key as an attachment to your email.' - - raise error_message if @mail_content.attachments.empty? - raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + raise 'NO_PUBKEY' if @mail_content.attachments.empty? + raise 'PUBKEY_NAME_ERR' unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ pub_key = @mail_content.attachments[0].body.decoded -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] mail-robot: add error code/message
by Luan Shengde 29 Dec '20

29 Dec '20

add error message for each error code before building the email_message. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 87 +++++++++++++++---- 1 file changed, 69 insertions(+), 18 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 593f345..242dfe1 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -3,32 +3,83 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_fail_email(my_info, message) - email_msg = <<~EMAIL_MESSAGE - To: #{my_info['my_email']} - Subject: [compass-ci] apply account failed +def email_err_message(message) + case message + when 'NO_COMMIT_URL' + err_message = <<~EMAIL_MESSAGE + No commit url found. + You should add a commit url for the 'apply account' email. - Dear user: + For example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'URL_PREFIX_ERR' + err_message = <<~EMAIL_MESSAGE + Please add a correct prefix for the commit url. + + my oss commit + + For example: + + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'NOT_REGISTERED' + err_message = <<~EMAIL_MESSAGE + Your repo has not been registered to our upstream-repos yet. + You should register your repo to the upstream first. + Try again after you have done it. + + Reference the following url to learn how to register the repo: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… + EMAIL_MESSAGE + when 'COMMIT_URL_NOT_AVAILABLE' + err_message = <<~EMAIL_MESSAGE + We cannot confirm the commit url matches your email. + + Make sure the commit exists. + Make sure it is truely submitted with your email. - Your application for account failed with following error: + Go to the commit history and find your commit. + Enter the commit and you will get your commit url. - #{message} + Try again after you have correctted the url + EMAIL_MESSAGE + when 'NO_PUBKEY' + err_message = <<~EMAIL_MESSAGE + No pub_key found. + Please add a pubkey to the email and then try again. + EMAIL_MESSAGE + when 'PUBKEY_NAME_ERR' + err_message = <<~EMAIL_MESSAGE + Pubkey file name error, keep its name as when it was generated. + The pubkey file name should like: - In order to successfully apply an account, please pay attention to the following points: + id_{{ xxx }}.pub + EMAIL_MESSAGE + end - 1. mail subject - The subject should exactly: apply account + err_message += <<~EMAIL_MESSAGE - 2. commit url - When you writing the url, add prefix: my oss commit - example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + Manual for how to apply account: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… + EMAIL_MESSAGE + + err_message +end + +def build_apply_account_fail_email(my_info, message) + err_message = email_err_message(message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: - attention: - Ensure you commit url exist and available to access. + Your application for account failed. - 3. ssh pubkey - You need to add a pubkey as an attachment to the email. + #{err_message} regards compass-ci -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] assign-account: change get to post
by Luan Shengde 29 Dec '20

29 Dec '20

use post is more easier to use rest-client to send request with data for applying account. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/assign-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/assign-account/assign-account.rb b/container/assign-account/assign-account.rb index 4ed95c0..c5d77db 100755 --- a/container/assign-account/assign-account.rb +++ b/container/assign-account/assign-account.rb @@ -12,9 +12,9 @@ require_relative 'get_account_info' set :bind, '0.0.0.0' set :port, 29999 -get '/assign_account' do +post '/assign_account' do begin - data = YAML.safe_load request.body.read + data = JSON.parse request.body.read rescue StandardError => e puts e.message puts e.backtrace -- 2.23.0

1 0

[PATCH v5 compass-ci] lib: add build_my_info_client
by Luan Shengde 29 Dec '20

29 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment, if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info/lab to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 76 +++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..162bf9f --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,76 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'nolab' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp, + 'my_login_name' => nil, + 'my_commit_url' => nil, + 'my_ssh_pubkey' => [] + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + default_yaml_info['lab'] = @lab + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

1 0

[PATCH compass-ci] container/auto-submit: add user lkp
by Li Yuanchao 29 Dec '20

29 Dec '20

[error message] auto-submit: whoami: unknown uid 1090 Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/auto-submit/Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/container/auto-submit/Dockerfile b/container/auto-submit/Dockerfile index 5b44551..52cd996 100644 --- a/container/auto-submit/Dockerfile +++ b/container/auto-submit/Dockerfile @@ -3,6 +3,7 @@ FROM alpine:3.11 +RUN adduser -u 1090 -D lkp RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories RUN apk update && \ -- 2.23.0

1 0

[PATCH compass-ci] lib/mail-post: add cc/bcc for send mail service
by Luan Shengde 29 Dec '20

29 Dec '20

enable user send Cc and Bcc email. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/mail-post.rb | 12 ++++++++++++ 1 file changed, 12 insertions(+) diff --git a/lib/mail-post.rb b/lib/mail-post.rb index ac455fa..9d3e601 100644 --- a/lib/mail-post.rb +++ b/lib/mail-post.rb @@ -20,6 +20,8 @@ post '/send_mail_yaml' do mail_info = { 'subject' => data['subject'], 'to' => data['to'], + 'cc' => data['cc'], + 'bcc' => data['bcc'], 'body' => data['body'] } @@ -32,6 +34,8 @@ post '/send_mail_text' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -45,6 +49,8 @@ post '/send_mail_encode' do mail_info = { 'subject' => data.subject, 'to' => data.to, + 'cc' => data.cc, + 'bcc' => data.bcc, 'body' => data.body.decoded } @@ -63,10 +69,16 @@ def send_mail(mail_info) from ENV['ROBOT_EMAIL_ADDRESS'] subject mail_info['subject'] to mail_info['to'] + cc mail_info['cc'] + bcc mail_info['bcc'] body mail_info['body'] end mail.deliver! + check_to_store_email(mail) +end + +def check_to_store_email(mail) return if ENV['SEND_MAIL_PORT'].to_s != '49000' return if ENV['HOST_SERVER'] != 'z9' -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/bisect: add PKGBUILD info in bisect report
by Cao Xueliang 29 Dec '20

29 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 7 +++++++ lib/mail_bisect_result.rb | 10 +++++++--- 2 files changed, 14 insertions(+), 3 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index f789a71..2043730 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -22,6 +22,7 @@ class GitBisect set_bad_job set_build_pkg_dir set_upstream + set_pkgbuild_repo set_work_dir set_bad_commit set_good_commit @@ -61,6 +62,11 @@ class GitBisect raise "checkout repo: #{@upstream_repo} to commit: #{@upstream_commit} failed!" unless @work_dir end + def set_pkgbuild_repo + @pkgbuild_repo = @bad_job['pkgbuild_repo'] + raise 'pkgbuild_repo is null' unless @pkgbuild_repo + end + def set_bad_commit @bad_commit = @upstream_commit end @@ -100,6 +106,7 @@ class GitBisect 'commit' => first_bad_commit, 'bisect_error' => bisect_error, 'all_errors' => all_errors, + 'pkgbuild_repo' => @pkgbuild_repo, 'first_bad_commit_result_root' => first_bad_commit_result_root] end diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 1d4ed69..5f88c36 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -9,10 +9,11 @@ require_relative 'assign_account_client' # compose and send email for bisect result class MailBisectResult def initialize(bisect_info) - @bisect_error = bisect_info['bisect_error'] - @all_errors = bisect_info['all_errors'] @repo = bisect_info['repo'] @commit_id = bisect_info['commit'] + @all_errors = bisect_info['all_errors'] + @bisect_error = bisect_info['bisect_error'] + @pkgbuild_repo = bisect_info['pkgbuild_repo'] @first_bad_commit_result_root = bisect_info['first_bad_commit_result_root'] @git_commit = GitCommit.new(@repo, @commit_id) # now send mail to review @@ -28,6 +29,7 @@ class MailBisectResult subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" prefix_srv = "http://#{ENV['SRV_HTTP_HOST']}:#{ENV['SRV_HTTP_PORT']}" bisect_job_url = ENV['result_root'] ? "bisect job info: #{prefix_srv}#{ENV['result_root']}\n" : '' + pkgbuild_repo_url = "PKGBUILD info: #{prefix_srv}/#{@pkgbuild_repo}\n" first_bad_commit_job_url = "first bad commit job info: #{prefix_srv}#{@first_bad_commit_result_root}\n" data = <<~BODY @@ -40,9 +42,11 @@ class MailBisectResult commit: #{@commit_id} ("#{(a)git_commit.subject}") compiler: gcc (GCC) 7.3.0 - all errors/warnings: + all errors/warnings (new ones prefixed by >>): #{@all_errors} + reference information: + #{pkgbuild_repo_url} #{bisect_job_url} #{first_bad_commit_job_url} Regards, -- 2.23.0

1 0

[PATCH v2 compass-ci] sparrow/READEME.md: fix format error
by Liu Yinsi 29 Dec '20

29 Dec '20

correct READEME format and content. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..c7a80e7 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - 执行上述命令正常情况下会提示信息如下： - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci搭建完毕，下面就可以开始进行测试了。 + 执行上述命令正常情况下会提示信息如下: + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci搭建完毕，执行步骤8下载所需要的rootfs文件就可以开始进行测试了。 8. 下载rootfs（以下载openeuler/aarch64/20.03为例，需要在哪个系统上测试就去对应的/srv/os/目录下使用wget命令下载cgz文件包） ```bash @@ -81,7 +81,9 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… id查看output文件 +14. [查看任务结果](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/br… + 本地搭建compass-ci用户可在srv/result目录下根据yaml名称/日期/$testbox/$os-$os-version-$os-arch/$job_id查看output文件(可用tab键自动补全多级目录方便查找) + 以查看iperf任务结果为例操如下： ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/nolab.1/output ``` -- 2.23.0

1 0

[PATCH compass-ci 3/3] auto_submit.yaml: add l/linux/linux for pkg_build
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkg_build need specific submit args for l/linux/linux Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index fce1dc0..69b73ff 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -3,6 +3,9 @@ a/AvxToNeon/AvxToNeon: c/compass-ci/compass-ci: - testbox=taishan200-2280-2s64p-256g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=taishan200-2280-2s64p-256g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml +l/linux/linux: +- testbox=vm-2p32g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-linux.yaml +- testbox=vm-2p32g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-linux.yaml archlinux: - testbox=vm-2p16g os=openeuler os_version=20.03 os_mount=nfs os_arch=aarch64 build-pkg.yaml - testbox=vm-2p16g os=archlinux os_version=2020-11-12 os_mount=nfs os_arch=aarch64 build-pkg.yaml -- 2.23.0

1 0

[PATCH compass-ci 2/3] auto_submit: add some log
by Li Yuanchao 29 Dec '20

29 Dec '20

Print git_repo to help find out if repo has been submitted job and the job id. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 1 + 1 file changed, 1 insertion(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index 4e26c87..f96be8c 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -42,6 +42,7 @@ class AutoSubmit def get_argvs(newrefs_info) git_repo = newrefs_info['git_repo'] + puts "git_repo : #{git_repo}" url = Array(newrefs_info['url'])[0] submit_argv = ["#{ENV['LKP_SRC']}/sbin/submit upstream_repo=#{git_repo} upstream_url=#{url}"] -- 2.23.0

1 0

[PATCH compass-ci 1/3] auto_submit: pkgbuild_repo may have specific submit args
by Li Yuanchao 29 Dec '20

29 Dec '20

Pkgbuild_repos all use the same submit args before. But some of them may need their own args, not the common one. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/auto_submit b/sbin/auto_submit index d765094..4e26c87 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -50,7 +50,7 @@ class AutoSubmit return unless pkgbuild_repo submit_argv.push("pkgbuild_repo=#{pkgbuild_repo}") - repo2job_key = 'archlinux' + repo2job_key = @repo2job[git_repo] ? git_repo : 'archlinux' else return unless @repo2job[git_repo] -- 2.23.0

1 0

[PATCH compass-ci] optimize READE.md
by Liu Yinsi 29 Dec '20

29 Dec '20

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/sparrow/README.md b/sparrow/README.md index a9a4bb0..4640974 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -52,10 +52,10 @@ submit iperf.yaml testbox=vm-2p8g ``` - �� - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory - compass-ci�� + �� + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + submit /c/lkp-tests/jobs/iperf.yaml failed, got job_id=0, error: Error resolving real path of '/srv/os/openeuler/aarch64/20.03/boot/vmlinuz': No such file or directory + compass-ci��8��rootfs�� 8. ��rootfs��openeuler/aarch64/20.03��/srv/os/��wget��cgz�� ```bash @@ -81,7 +81,8 @@ cd /c/compass-ci/providers/ && ./my-qemu.sh ``` -14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md),��compass-ci��srv/result��job id��output�� +14. [��](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/browse-results.zh.md) ```bash - cd /srv/result/iperf/$job_id/output + cd /srv/result/iperf/2020-12-29/vm-2p8g/openeuler-20.09-aarch64/$job_id/output ``` + ��compass-ci��srv/result��yaml��/��/testbox/os-os-version-os-arch/job_id��output��(��tab��) -- 2.23.0

1 1

[PATCH v2 compass-ci] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

1.some comment and code is unnecessary, so remove. 2.put `echo **` together better. test: /srv/result/host-info/2020-12-29/vm-2p8g/openeuler-20.09-aarch64 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..fd921dc 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -18,14 +18,9 @@ fi qemu=qemu-system-aarch64 command -v $qemu >/dev/null || qemu=qemu-kvm -echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= @@ -47,8 +42,6 @@ do *) ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -63,6 +56,7 @@ done < $ipxe_script initrd=initrd cat $initrds > $initrd +echo $SCHED_PORT echo kernel: $kernel echo initrds: $initrds echo append: $append @@ -78,7 +72,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: get field state count
by Lu Kaiyi 29 Dec '20

29 Dec '20

[why] get field state count for overview of report page. [input] suite/os/tbox_group/... eg: suite [output] {"build-pkg"=> {"failed"=>5543, "finished"=>906, "incomplete"=>197, "submit"=>136, "OOM"=>11, "post_run"=>2}, "cci-depends"=> {"finished"=>1782, "failed"=>675, "incomplete"=>279, "post_run"=>3, "OOM"=>1, "running"=>1}, "cci-makepkg"=> {"failed"=>1659, "finished"=>1231, "incomplete"=>384, "OOM"=>4, "post_run"=>1}, "iperf"=>{"finished"=>179, "failed"=>94}, "spinlock"=>{"finished"=>52, "failed"=>1}, "iperf-walk-os-test"=>{"finished"=>172, "failed"=>24, "OOM"=>1}, "stream"=>{"finished"=>242, "incomplete"=>157, "failed"=>7}, "deploy-cci"=>{"finished"=>27, "failed"=>19}, "borrow"=>{"finished"=>101, "incomplete"=>12, "failed"=>4}, "bisect"=>{"finished"=>169, "failed"=>151}, ... } Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/web_backend.rb | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index ea67064..15e2ad0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -509,3 +509,37 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def query_field_state(field) + query = { + aggs: { + "all_#{field}" => { + terms: { field: field, size: 1000 }, + aggs: { + all_state: { + terms: { field: 'job_state' } + } + } + } + }, + size: 0 + } + es_result = es_query(query)['aggregations']["all_#{field}"]['buckets'] + es_result.sort_by! { |h| h['doc_count'] } + es_result.reverse! + + parse_field_state(es_result) +end + +def parse_field_state(es_result) + result_hash = {} + es_result.each do |result| + key = result['key'] + all_state = result['all_state']['buckets'] + state_count = {} + all_state.map { |state| state_count[state['key']] = state['doc_count'] } + result_hash[key] = state_count unless state_count.empty? + end + + result_hash +end -- 2.23.0

2 1

[PATCH compass-ci] docker-rootfs: add README.md
by Wang Chenglong 29 Dec '20

29 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..5becfd0 --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,17 @@ +# Use the tool to create a new nfsroot for os + +Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs + +Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + +Some configuration items: + ./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. -- 2.23.0

3 3

[PATCH compass-ci 2/3] providers/qemu: encapsulated into functions
by Liu Yinsi 29 Dec '20

29 Dec '20

test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272000 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 171 +++++++++++++++++++++++++----------------- 1 file changed, 104 insertions(+), 67 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index f84c5ae..48bacab 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -7,74 +7,111 @@ : ${nr_cpu:=1} : ${memory:=1G} -log_file=/srv/cci/serial/logs/${hostname} -if [ ! -f "$log_file" ]; then - touch $log_file - # fluentd refresh time is 1s - # let fluentd to monitor this file first - sleep 2 -fi - -qemu=qemu-system-aarch64 -command -v $qemu >/dev/null || qemu=qemu-kvm - -echo $SCHED_PORT -ipxe_script=ipxe_script -curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script -cat $ipxe_script >> ${log_file} - -append= -initrds= -while read a b c -do - case "$a" in - initrd) - file=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - initrds+="$file " - ;; - kernel) - kernel=$(basename "$b") - wget --timestamping -a ${log_file} --progress=bar:force $b - append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") - ;; - esac -done < $ipxe_script - -[ -s "$kernel" ] || { - echo "can't get kernel or kernel size is 0" - exit +check_logfile() +{ + log_file=/srv/cci/serial/logs/${hostname} + if [ ! -f "$log_file" ]; then + touch $log_file + # fluentd refresh time is 1s + # let fluentd to monitor this file first + sleep 2 + fi } -[ -n "$initrds" ] || { - exit +qemu_command() +{ + qemu=qemu-system-aarch64 + command -v $qemu >/dev/null || qemu=qemu-kvm } -initrd=initrd -cat $initrds > $initrd - -echo kernel: $kernel -echo initrds: $initrds -echo append: $append -echo less $log_file - -sleep 5 - -kvm=( - $qemu - -machine virt-4.0,accel=kvm,gic-version=3 - -kernel $kernel - -initrd $initrd - -smp $nr_cpu - -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} - -k en-us - -no-reboot - -nographic - -serial file:${log_file} - -monitor null -) - -"${kvm[@]}" --append "${append}" +write_logfile() +{ + ipxe_script=ipxe_script + curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script + cat $ipxe_script >> ${log_file} +} + +get_rootfs() +{ + append= + initrds= + while read a b c + do + case "$a" in + initrd) + file=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + initrds+="$file " + ;; + kernel) + kernel=$(basename "$b") + wget --timestamping -a ${log_file} --progress=bar:force $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + esac + done < $ipxe_script +} + +check_rootfs() +{ + [ -s "$kernel" ] || { + echo "can't get kernel or kernel size is 0" + exit + } + + [ -n "$initrds" ] || { + exit + } + + initrd=initrd + cat $initrds > $initrd +} + +print_message() +{ + echo $SCHED_PORT + echo kernel: $kernel + echo initrds: $initrds + echo append: $append + echo less $log_file + + sleep 5 +} + +kvm_command() +{ + kvm=( + $qemu + -machine virt-4.0,accel=kvm,gic-version=3 + -kernel $kernel + -initrd $initrd + -smp $nr_cpu + -m $memory + -cpu Kunpeng-920 + -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd + -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} + -k en-us + -no-reboot + -nographic + -serial file:${log_file} + -monitor null + ) +} + +run_qemu() +{ + "${kvm[@]}" --append "${append}" +} + +check_logfile +write_logfile + +get_rootfs +check_rootfs + +qemu_command + +print_message + +kvm_command +run_qemu -- 2.23.0

2 2

[PATCH compass-ci 1/3] providers/qemu: delete redundant code
by Liu Yinsi 29 Dec '20

29 Dec '20

some comment and code is unnecessary, so remove. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.271979 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index fe2345f..f84c5ae 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -22,18 +22,12 @@ echo $SCHED_PORT ipxe_script=ipxe_script curl http://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/boot.ipxe/mac/${mac} > $ipxe_script cat $ipxe_script >> ${log_file} -#echo ----- -#cat $ipxe_script -#echo ----- -#exit append= initrds= while read a b c do case "$a" in - '#') - ;; initrd) file=$(basename "$b") wget --timestamping -a ${log_file} --progress=bar:force $b @@ -44,11 +38,7 @@ do wget --timestamping -a ${log_file} --progress=bar:force $b append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") ;; - *) - ;; esac -#done < /tftpboot/boot.ipxe-debian -#done < /tftpboot/boot.ipxe-centos done < $ipxe_script [ -s "$kernel" ] || { @@ -78,7 +68,6 @@ kvm=( -smp $nr_cpu -m $memory -cpu Kunpeng-920 - -device virtio-gpu-pci -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -- 2.23.0

2 4

[PATCH v3 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..5572274 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,10 @@ dirs=( ) mkdir -p "${dirs[@]}" + +if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd +elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr +fi -- 2.23.0

1 0

[PATCH compass-ci] docker-rootfs/common: add comment for set password file.
by Wang Chenglong 29 Dec '20

29 Dec '20

[Why] add password file is easy to ignore. Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/common | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/docker-rootfs/common b/container/docker-rootfs/common index fe37da9..2ee16c8 100755 --- a/container/docker-rootfs/common +++ b/container/docker-rootfs/common @@ -33,7 +33,8 @@ usage() dst_rootfs_new_abspath: destination absolute path to create for rootfs Example: - ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + ./run /tmp/openEuler-docker/openEuler-docker.${aarch}.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists " } -- 2.23.0

1 0

[PATCH v2 compass-ci] docker-rootfs: add README.md
by Wang Chenglong 29 Dec '20

29 Dec '20

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/docker-rootfs/README.md | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) create mode 100644 container/docker-rootfs/README.md diff --git a/container/docker-rootfs/README.md b/container/docker-rootfs/README.md new file mode 100644 index 0000000..e95f55b --- /dev/null +++ b/container/docker-rootfs/README.md @@ -0,0 +1,18 @@ +# Use the tool to create a new nfsroot for os + +Usage: + ./run <src_docker_file_abspath> <dst_rootfs_new_abspath> + + src_docker_file_abspath: source .tar.xz file absolute path with suffix: [ tar.xz ]. + dst_rootfs_new_abspath: destination absolute path to create for rootfs. + +Example: + ./run /tmp/openEuler-docker/openEuler-docker.aarch64.tar.xz /tmp/openeuler-rootfs/ + # please ensure $HOME/.config/compass-ci/rootfs-passwd exists + +Some configuration items: + ./packages-to-install + If you want to pre-install the software, you can write the package names in ./packages-to-install. + + $HOME/.config/compass-ci/rootfs-passwd + Set the password for the image into this file. -- 2.23.0

1 0

[PATCH v2 compass-ci 2/2] sbin/es-find: add -d/--desc-keyword option
by Lu Weitao 29 Dec '20

29 Dec '20

es-find support order result by keyword with desc [Example] es-find suite=iperf -d/--desc-keyword start_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 10 +++++++--- 1 file changed, 7 insertions(+), 3 deletions(-) diff --git a/sbin/es-find b/sbin/es-find index 2397338..17eefe6 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -29,8 +29,8 @@ def parse_argv items end -options = {} -options[:nr_jobs] = 10 +options = { nr_jobs: 10, desc_keyword: nil } + opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -45,6 +45,10 @@ opt_parser = OptionParser.new do |opts| options[:nr_jobs] = n.to_i end + opts.on('-d', '--desc-keyword KEYWORD', 'order by keyword: start_time|end_time|...') do |d| + options[:desc_keyword] = d + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -54,6 +58,6 @@ end opt_parser.parse!(ARGV) items = parse_argv es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs]) +query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

1 0

[PATCH v2 compass-ci 1/2] lib/es_query.rb: support order by keyword with desc
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] user need get latest doc in ES [Sample] caller(es-find | compare | web-backend/compare | ...) use like: es = ESQuery.new(index: 'jobs') query_result = es.multi_field_query(items, desc_keyword: 'start_time') Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..10b9644 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,7 +18,7 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000) + def multi_field_query(items, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit @@ -31,6 +31,7 @@ class ESQuery } }, size: size } + query.merge!(assign_desc_body(desc_keyword)) if desc_keyword @client.search index: @index + '*', body: query end @@ -112,3 +113,11 @@ def parse_conditions(items) end items_hash end + +def assign_desc_body(keyword) + { + sort: [{ + keyword => { order: 'desc' } + }] + } +end -- 2.23.0

1 0

[PATCH v2 compass-ci] lib/params_group.rb optimize filter_groups()
by Lu Weitao 29 Dec '20

29 Dec '20

[Why] 1. We had for-each key-value in hash, needn't query value by key once more. 2. When remove empty value of Hash(String, Hash(String, Array(...)), remove the first level key-value at first. Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..fa71653 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,13 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + if value.empty? + groups.delete(group_key) + next + end + value.each do |dim_key, job_list| + value.delete(dim_key) if job_list.empty? end - groups.delete(group_key) if groups[group_key].empty? end end -- 2.23.0

1 0

[PATCH v2 compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 29 Dec '20

29 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..538f45d 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,13 @@ dirs=( ) mkdir -p "${dirs[@]}" + +[ -n "$os_mount" ] && { + if [ "$os_mount" = "nfs" ]; then + mount -t "$os_mount" $LKP_SERVER:os /srv/os + mount -t "$os_mount" $LKP_SERVER:initrd /srv/initrd + elif [ "$os_mount" = "cifs" ]; then + mount -t "$os_mount" //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr + mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr + fi +} -- 2.23.0

4 3

[PATCH lkp-tests] INSTALL.md: add quick installation description
by Wei Jihui 29 Dec '20

29 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- INSTALL.md | 21 +++++++++++++++++---- 1 file changed, 17 insertions(+), 4 deletions(-) diff --git a/INSTALL.md b/INSTALL.md index 1bf0acc7..d7ce3517 100644 --- a/INSTALL.md +++ b/INSTALL.md @@ -1,18 +1,30 @@ +# Quick installation +```shell + git clone https://gitee.com/wu_fengguang/lkp-tests.git + cd lkp-tests + make install + source $HOME/.${SHELL##*/}rc +``` +all is ok! + +# Step-by-Step installation + system wide setup ================= - ## debian - +```shell sudo apt-get install ruby-git ruby-activesupport +``` ## openEuler - +```shell sudo dnf install ruby rubygems gem install git activesupport +``` per-user setup ============== - +```shell git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests echo "export LKP_SRC=$PWD" >> $HOME/.${SHELL##*/}rc @@ -24,3 +36,4 @@ per-user setup hdd_partitions: ssd_partitions: EOF +``` -- 2.23.0

1 0

[PATCH v3 compass-ci 3/3] scheduler: idle-job: submit idle job when finding no job
by Ren Wen 29 Dec '20

29 Dec '20

There're 2 types of queues to find job in order: 1. user queue: 'sched/<taishan-a12>/<user-email>' 2. idle queue: 'sched/<taishan-a12>/idle' Auto submit idle jobs when finding no job. This feature is for physical machines. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/find_job_boot.cr | 25 ++++++++++++++++++------- 1 file changed, 18 insertions(+), 7 deletions(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 5f71b32..21396c4 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -53,25 +53,33 @@ class Sched end def get_queues(host) - queues = [] of String - queues_str = @redis.hash_get("sched/host2queues", host) - return queues unless queues_str + return [] of String unless queues_str + default_queues = [] of String queues_str.split(',', remove_empty: true) do |item| - queues << item.strip + default_queues << item.strip end sub_queues = [] of String - queues.each do |queue| - matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + default_queues.each do |queue| + matched_queues = @redis.keys("#{QUEUE_NAME_BASE}/sched/#{queue}/*/ready") + next if matched_queues.empty? + matched_queues.each do |mq| match_data = "#{mq}".match(%r(^#{QUEUE_NAME_BASE}/sched/(#{queue}/.+)/ready$)) sub_queues << match_data[1] if match_data end end - return rand_queues(sub_queues) + return sub_queues unless sub_queues.empty? + + idle_queues = [] of String + default_queues.each do |queue| + idle_queues << "#{queue}/idle" + end + + return idle_queues end def get_job_from_queues(queues, testbox) @@ -91,6 +99,9 @@ class Sched if job create_job_cpio(job.dump_to_json_any, Kemal.config.public_folder) + else + # for physical machines + spawn { auto_submit_idle_job(host) } end return boot_content(job, boot_type) -- 2.23.0

1 0

[PATCH v3 compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 29 Dec '20

29 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v3 compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 29 Dec '20

29 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use 'my_email' instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci 6/6] mail-robot: optimize parse email_file
by Luan Shengde 29 Dec '20

29 Dec '20

[why] avoid build too much messages in the code files [how] just raise error code in the code file, and build the error message in the file building email message Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../mail-robot/lib/gitee-commit-url-check.rb | 4 +-- .../lib/parse-apply-account-email.rb | 30 +++++++------------ 2 files changed, 11 insertions(+), 23 deletions(-) diff --git a/container/mail-robot/lib/gitee-commit-url-check.rb b/container/mail-robot/lib/gitee-commit-url-check.rb index 7cac0d5..af31e17 100755 --- a/container/mail-robot/lib/gitee-commit-url-check.rb +++ b/container/mail-robot/lib/gitee-commit-url-check.rb @@ -38,9 +38,7 @@ class GiteeCommitUrlCheck def gitee_commit_exist(email_index) return if email_index - error_message = "We can not confirm whether the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with your email.' + error_message = 'COMMIT_URL_NOT_AVAILABLE' raise error_message end diff --git a/container/mail-robot/lib/parse-apply-account-email.rb b/container/mail-robot/lib/parse-apply-account-email.rb index 7024731..94becdf 100755 --- a/container/mail-robot/lib/parse-apply-account-email.rb +++ b/container/mail-robot/lib/parse-apply-account-email.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative 'gitee-commit-url-check' # used to parse mail_content for my_commit_url and my_ssh_pubkey @@ -27,7 +28,7 @@ require_relative 'gitee-commit-url-check' # gitee.com: # call GiteeCommitUrlCheck to check the commit # non-gitee.com: -# execute curl to check the commit +# get request date for the commit url to check the commit # parse_pub_key # check my_ssh_pubkey exists and return it class ParseApplyAccountEmail @@ -63,10 +64,8 @@ class ParseApplyAccountEmail # the commit url should be in a standart format, example: # my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… unless mail_content_line.match?(%r{my oss commit:\s*https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) - error_message = "No matched commit url found.\n" - error_message += 'Ensure that you have add a right commit url, ' - error_message += "and with prefix 'my oss commit:'." - raise error_message + raise 'NO_COMMIT_URL' unless mail_content_line.match?(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}}) + raise 'URL_PREFIX_ERR' unless mail_content_line.match?(%r{my oss commit:\s*https?://}) end mail_content_line.match(%r{https?://[^/]*/[^/]*/[^/]*/commit/[\w\d]{40}})[0] @@ -88,16 +87,13 @@ class ParseApplyAccountEmail return unless match_out.empty? - error_message = "The repo url for your commit is not in our upstream-repo list.\n" - error_message += 'Use a new one, or consulting the manager for available repo list.' - - raise error_message + raise 'NOT_REGISTERED' end def commit_url_availability(url, base_url) hub_name = url.split('/')[2] - # it requires authentication when execute curl to get the commit information + # it requires authentication when sending request to get the commit information # clone the repo and then validate the commit for the email address if hub_name.eql? 'gitee.com' gitee_commit(url, base_url) @@ -112,24 +108,18 @@ class ParseApplyAccountEmail end def non_gitee_commit(url) - url_fdback = %x(curl #{url}) + url_fdback = RestClient.get(url).body email_index = url_fdback.index @my_info['my_email'] return unless email_index.nil? - error_message = "We can not confirm the commit url matches your email.\n" - error_message += 'Make sure that the commit url is right,' - error_message += ' or it is truely submitted with you email.' - - raise error_message + raise 'COMMIT_URL_NOT_AVAILABLE' end def parse_pub_key - error_message = "No pub_key found.\n" - error_message += 'Please add a pub_key as an attachment to your email.' - raise error_message if @mail_content.attachments.empty? - raise error_message unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ + raise 'NO_PUBKEY' if @mail_content.attachments.empty? + raise 'PUBKEY_NAME_ERR' unless @mail_content.attachments[0].filename =~ /^id_.*\.pub$/ pub_key = @mail_content.attachments[0].body.decoded -- 2.23.0

2 2

[PATCH v4 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 29 Dec '20

29 Dec '20

[why] systemctl is not supported when use docker Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..c2404dbfd 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -5,20 +5,35 @@ # sshr_port_len # sshr_port_base -. $LKP_SRC/lib/http.sh +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci 4/4] assign-account: add 'lab' to default yaml
by Luan Shengde 29 Dec '20

29 Dec '20

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/get_account_info.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/assign-account/get_account_info.rb b/container/assign-account/get_account_info.rb index af08d91..7e0a5a7 100755 --- a/container/assign-account/get_account_info.rb +++ b/container/assign-account/get_account_info.rb @@ -173,6 +173,7 @@ class AccountStorage # they are added along with 'my_ssh_pubkey' when sending assign account request account_yaml['my_email'] = @data['my_email'] account_yaml['my_name'] = @data['my_name'] + account_yaml['lab'] = @data['lab'] File.open(default_yaml, 'w') do |f| f.puts account_yaml.to_yaml -- 2.23.0

2 2

[PATCH v4 compass-ci] lib: add build_my_info_client
by Luan Shengde 29 Dec '20

29 Dec '20

purpose: build my_info for users when they installing compass-ci as private environment, if not, will face account verification problems when they submitting jobs - store my_info to ES - write my_info/lab to default/lab yaml file Signed-off-by: Luan Shengde <shdluan(a)163.com> --- lib/build_my_info_client.rb | 76 +++++++++++++++++++++++++++++++++++++ 1 file changed, 76 insertions(+) create mode 100755 lib/build_my_info_client.rb diff --git a/lib/build_my_info_client.rb b/lib/build_my_info_client.rb new file mode 100755 index 0000000..97ad757 --- /dev/null +++ b/lib/build_my_info_client.rb @@ -0,0 +1,76 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'fileutils' +require 'io/console' +require_relative '../lib/es_client' + +# purpose: +# build my_info +# store my_info to ES +# write my_info to default/lab yaml file +# for user install compass-ci at their local server +# user meet verification problems when submit jobs +# add calling in installing script to meet the verification function +# usage: +# require_relative 'build_my_info_client' +# +# build_my_info = BuildMyInfo.new(email, name, lab) +# build_my_info.config_my_info +class BuildMyInfo + def initialize(my_email, my_name, lab) + @lab = lab || 'z9' + @my_info = { + 'my_email' => my_email, + 'my_name' => my_name, + 'my_token' => %x(uuidgen).chomp, + 'my_login_name' => nil, + 'my_commit_url' => nil, + 'my_ssh_pubkey' => [] + } + end + + def config_default_yaml + default_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/defaults" + FileUtils.mkdir_p default_yaml_dir unless File.directory? default_yaml_dir + default_yaml_file = "#{default_yaml_dir}/account.yaml" + FileUtils.touch(default_yaml_file) unless File.exist? default_yaml_file + + default_yaml_info = YAML.load_file(default_yaml_file) || {} + default_yaml_info['my_email'] = @my_info['my_email'] + default_yaml_info['my_name'] = @my_info['my_name'] + default_yaml_info['lab'] = @lab + + File.open(default_yaml_file, 'w') do |f| + f.puts default_yaml_info.to_yaml + end + end + + def config_lab_yaml + lab_yaml_dir = "#{ENV['HOME']}/.config/compass-ci/include/lab" + FileUtils.mkdir_p lab_yaml_dir unless File.directory? lab_yaml_dir + lab_yaml_file = "#{lab_yaml_dir}/#{(a)lab}.yaml" + FileUtils.touch(lab_yaml_file) unless File.exist? lab_yaml_file + + lab_yaml_info = YAML.load_file(lab_yaml_file) || {} + lab_yaml_info['my_token'] = @my_info['my_token'] + + File.open(lab_yaml_file, 'w') do |f| + f.puts lab_yaml_info.to_yaml + end + end + + def store_account_info + es = ESClient.new(index: 'accounts') + es.put_source_by_id(@my_info['my_email'], @my_info) + end + + def config_my_info + config_default_yaml + config_lab_yaml + store_account_info + end +end -- 2.23.0

2 2

[PATCH compass-ci 2/2] sbin/es-find: add -d/--desc-keyword option
by Lu Weitao 29 Dec '20

29 Dec '20

es-find support order result by keyword with desc [Example] es-find suite=iperf -d/--desc-keyword start_time Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-find | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/sbin/es-find b/sbin/es-find index 2397338..985fdea 100755 --- a/sbin/es-find +++ b/sbin/es-find @@ -31,6 +31,8 @@ end options = {} options[:nr_jobs] = 10 +options[:desc_keyword] = nil + opt_parser = OptionParser.new do |opts| opts.banner = 'Usage: es-find [options] search_key1=val1[,val2..] ..' @@ -45,6 +47,10 @@ opt_parser = OptionParser.new do |opts| options[:nr_jobs] = n.to_i end + opts.on('-d', '--desc-keyword KEYWORD', 'order by keyword: start_time|end_time|...') do |d| + options[:desc_keyword] = d + end + opts.on_tail('-h', '--help', 'show this message') do puts opts exit @@ -54,6 +60,6 @@ end opt_parser.parse!(ARGV) items = parse_argv es = ESQuery.new(ES_HOST, ES_PORT) -query_result = es.multi_field_query(items, size: options[:nr_jobs]) +query_result = es.multi_field_query(items, size: options[:nr_jobs], desc_keyword: options[:desc_keyword]) json_string = JSON.pretty_generate(query_result) puts json_string -- 2.23.0

2 2

[PATCH v2 compass-ci] sparrow/5-build: adapt for x86_64 machine
by Liu Yinsi 29 Dec '20

29 Dec '20

build ipxe according different arch. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/5-build/ipxe | 30 ++++++++++++++++++++++-------- 1 file changed, 22 insertions(+), 8 deletions(-) diff --git a/sparrow/5-build/ipxe b/sparrow/5-build/ipxe index a66e298..50a2af9 100755 --- a/sparrow/5-build/ipxe +++ b/sparrow/5-build/ipxe @@ -15,11 +15,25 @@ cd ipxe/src || exit cpus=$(nproc) -make ARCH=arm64 bin-arm64-efi/ipxe.efi -make ARCH=arm64 bin-arm64-efi/snponly.efi -make ARCH=arm64 bin-arm64-efi/snp.efi -make CONFIG=rpi bin-arm64-efi/rpi.efi - -mkdir -p /tftpboot/ipxe/bin-arm64-efi/ -cp -a bin-arm64-efi/*.efi /tftpboot/ipxe/bin-arm64-efi/ -chmod 664 /tftpboot/ipxe/bin-arm64-efi/*.efi +build_ipxe() +{ + make ARCH=$1 bin-$1-efi/ipxe.efi + make ARCH=$1 bin-$1-efi/snponly.efi + make ARCH=$1 bin-$1-efi/snp.efi + make CONFIG=rpi bin-$1-efi/rpi.efi + + mkdir -p /tftpboot/ipxe/bin-$1-efi/ + cp -a bin-$1-efi/*.efi /tftpboot/ipxe/bin-$1-efi/ + chmod 664 /tftpboot/ipxe/bin-$1-efi/*.efi +} + +ARCH=$(arch) + +case "$ARCH" in + "aarch64") + build_ipxe arm64 + ;; + "x86_64") + build_ipxe x86_64 + ;; +esac -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] container: mount "/srv/git/archlinux" to srv-http
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/srv-http/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start b/container/srv-http/start index 8c8ca0b..d9f461b 100755 --- a/container/srv-http/start +++ b/container/srv-http/start @@ -15,6 +15,7 @@ cmd=( -v /srv/result:/srv/result:ro -v /srv/initrd/qemu-image:/srv/initrd/qemu-image:ro -v /srv/os:/srv/os:ro + -v /srv/git/archlinux:/srv/git/archlinux:ro -d srv-http ) -- 2.23.0

1 0

[PATCH v2 compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 0

[PATCH v2 compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..47ef92f --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,15 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +MAINTAINER caoxl78320(a)163.com + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v2 compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..a1dbe77 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load_file(File.join('/srv', "#{result_root}", 'job.yaml')) +end -- 2.23.0

1 0

[PATCH compass-ci] container/qemu-efi: adapt for x86_64
by Liu Yinsi 28 Dec '20

28 Dec '20

in x86_64 debian machine, UEFI support for qemu is ovmf/OVMF.fd, different arch use different UEFI support. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/qemu-efi/Dockerfile | 4 +++- container/qemu-efi/build | 5 ++++- container/qemu-efi/install | 6 +++++- 3 files changed, 12 insertions(+), 3 deletions(-) diff --git a/container/qemu-efi/Dockerfile b/container/qemu-efi/Dockerfile index e1087c5..46f442e 100644 --- a/container/qemu-efi/Dockerfile +++ b/container/qemu-efi/Dockerfile @@ -3,9 +3,11 @@ FROM debian +ARG UEFI + ENV DEBIAN_FRONTEND noninteractive COPY conf/sources.list* /etc/apt/ RUN apt-get update && \ apt-get install -y --no-install-recommends apt-utils > /dev/null 2>&1 && \ - apt-get install -y qemu-efi-aarch64 + apt-get install -y $UEFI diff --git a/container/qemu-efi/build b/container/qemu-efi/build index 1238e67..cac9426 100755 --- a/container/qemu-efi/build +++ b/container/qemu-efi/build @@ -2,4 +2,7 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. -docker build -t debian:qemu-efi . +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker build -t debian:qemu-efi --build-arg UEFI=$uefi . diff --git a/container/qemu-efi/install b/container/qemu-efi/install index 96be9aa..04bf581 100755 --- a/container/qemu-efi/install +++ b/container/qemu-efi/install @@ -11,5 +11,9 @@ cmd=( ) "${cmd[@]}" -docker cp qemu-efi:/usr/share/qemu-efi-aarch64 /usr/share/qemu-efi-aarch64 + +[ $(arch) == "aarch64" ] && uefi=qemu-efi-aarch64 +[ $(arch) == "x86_64" ] && uefi=ovmf + +docker cp qemu-efi:/usr/share/$uefi /usr/share/$uefi docker_rm qemu-efi -- 2.23.0

2 2

[PATCH lkp-tests] distro/depends/nginx-server: spelling error
by Wei Jihui 28 Dec '20

28 Dec '20

Signed-off-by: Wei Jihui <weijihuiall(a)163.com> --- distro/depends/nginx-server | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/distro/depends/nginx-server b/distro/depends/nginx-server index da35d5f6..85e9cc2e 100644 --- a/distro/depends/nginx-server +++ b/distro/depends/nginx-server @@ -1,6 +1,6 @@ gcc glibc -ethool +ethtool libssl-dev zlib1g-dev libpcre3-dev -- 2.23.0

1 0

[PATCH compass-ci 3/3] providers/qemu: adapt kvm.sh for x86_64 and riscv64
by Liu Yinsi 28 Dec '20

28 Dec '20

put same options in public_options, different options in individual_options, merge options according different arch. test result: /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-aarch64/z9.272042 /srv/result/host-info/2020-12-28/vm-2p8g/openeuler-20.09-x86_64/z9.272046 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-riscv64/z9.272055 /srv/result/host-info/2020-12-28/vm-2p8g/debian-sid-x86_64/z9.272057 Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- providers/qemu/kvm.sh | 78 ++++++++++++++++++++++++++++++++++++++----- 1 file changed, 70 insertions(+), 8 deletions(-) diff --git a/providers/qemu/kvm.sh b/providers/qemu/kvm.sh index 48bacab..83b9606 100755 --- a/providers/qemu/kvm.sh +++ b/providers/qemu/kvm.sh @@ -20,8 +20,19 @@ check_logfile() qemu_command() { - qemu=qemu-system-aarch64 - command -v $qemu >/dev/null || qemu=qemu-kvm + qemu=( + qemu-system-aarch64 + qemu-system-x86_64 + qemu-kvm + ) + + for qemu in "${qemu[@]}" + do + [ -n "$(command -v ${qemu})" ] && break + done + + # debian has both qemu-system-x86_64 and qemu-system-riscv64 command + [ -n "$(echo $kernel | grep "riscv64")" ] && qemu=qemu-system-riscv64 } write_logfile() @@ -31,6 +42,17 @@ write_logfile() cat $ipxe_script >> ${log_file} } +options_value() +{ + bios=/usr/share/qemu-efi-aarch64/QEMU_EFI.fd + ls $bios >/dev/null 2>&1 || bios=/usr/share/ovmf/OVMF.fd + + helper=/usr/libexec/qemu-bridge-helper + ls $helper >/dev/null 2>&1 || helper=/usr/lib/qemu/qemu-bridge-helper + + nic="tap,model=virtio-net-pci,helper=$helper,br=br0,mac=${mac}" +} + get_rootfs() { append= @@ -78,18 +100,14 @@ print_message() sleep 5 } -kvm_command() +public_options() { kvm=( $qemu - -machine virt-4.0,accel=kvm,gic-version=3 -kernel $kernel -initrd $initrd -smp $nr_cpu -m $memory - -cpu Kunpeng-920 - -bios /usr/share/qemu-efi-aarch64/QEMU_EFI.fd - -nic tap,model=virtio-net-pci,helper=/usr/libexec/qemu-bridge-helper,br=br0,mac=${mac} -k en-us -no-reboot -nographic @@ -98,6 +116,46 @@ kvm_command() ) } +individual_options() +{ + aarch64=( + -machine virt-4.0,accel=kvm,gic-version=3 + -cpu Kunpeng-920 + -bios $bios + -nic $nic + ) + + x86_64=( + -bios $bios + -nic $nic + ) + + riscv64=( + -machine virt + -device virtio-net-device,netdev=net0,mac=${mac} + -netdev bridge,br=br0,id=net0,helper=${helper} + ) +} + +merge_options() +{ + case $qemu in + qemu-system-aarch64) + kvm=("${kvm[@]}" "${aarch64[@]}") + ;; + qemu-system-x86_64) + kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-kvm) + [ "$(arch)" == "aarch64" ] && kvm=("${kvm[@]}" "${aarch64[@]}") + [ "$(arch)" == "x86_64" ] && kvm=("${kvm[@]}" "${x86_64[@]}") + ;; + qemu-system-riscv64) + kvm=("${kvm[@]}" "${riscv64[@]}") + ;; + esac +} + run_qemu() { "${kvm[@]}" --append "${append}" @@ -110,8 +168,12 @@ get_rootfs check_rootfs qemu_command +options_value print_message -kvm_command +public_options +individual_options +merge_options + run_qemu -- 2.23.0

1 0

[PATCH v2 compass-ci 2/3] scheduler: idle-job: hand over idle job
by Ren Wen 28 Dec '20

28 Dec '20

When submitting an idle job, the job will be submitted to "queues/sched/<queue>/idle"; When consuming an idle job, the job will be moved to "queues/sched/<queue>/in_process"; It will hand over job from "queues/sched/<queue>/idle/in_process" before. It's wrong for idle job, so fix it. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/close_job.cr | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/scheduler/close_job.cr b/src/scheduler/close_job.cr index 9b4da6a..bce6b33 100644 --- a/src/scheduler/close_job.cr +++ b/src/scheduler/close_job.cr @@ -14,8 +14,11 @@ class Sched raise "es set job content fail!" end + subqueue = job.subqueue + queue = (subqueue == "idle" ? job.queue : "#{job.queue}/#{subqueue}") + response = @task_queue.hand_over_task( - "sched/#{job.queue}/#{job.subqueue}", "extract_stats", job_id + "sched/#{queue}", "extract_stats", job_id ) if response[0] != 201 raise "#{response}" -- 2.23.0

1 0

[PATCH v2 compass-ci 1/3] scheduler: idle-job: fix queue when submitting job
by Ren Wen 28 Dec '20

28 Dec '20

[Before]: 1) job has a field 'idle_job' when submitting idle job. 2) field 'uuid' is used with remote user to do a rate limit. [Now]: 1) idle job has a subqueue 'idle'. 2) use 'my_email' instead of 'uuid'. Signed-off-by: Ren Wen <15991987063(a)163.com> --- src/scheduler/submit_job.cr | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/scheduler/submit_job.cr b/src/scheduler/submit_job.cr index c322d0f..c09d145 100644 --- a/src/scheduler/submit_job.cr +++ b/src/scheduler/submit_job.cr @@ -114,13 +114,6 @@ class Sched def submit_single_job(job) queue = "#{job.queue}/#{job.subqueue}" - # only single job will has "idle job" and "execute rate limiter" - if job["idle_job"].empty? - queue += "#{job.get_uuid_tag}" - else - queue = "#{queue}/idle" - end - job_id = add_task(queue, job.lab) return [{ "job_id" => "0", -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/es_query.rb: support order by keyword with desc
by Lu Weitao 28 Dec '20

28 Dec '20

[Why] user need get latest doc in ES [Sample] caller(es-find | compare | web-backend/compare | ...) use like: es = ESQuery.new(index: 'jobs') query_result = es.multi_field_query(items, desc_keyword: 'start_time') Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/es_query.rb | 11 ++++++++++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 3bea624..10b9644 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -18,7 +18,7 @@ class ESQuery # Example @items: { key1 => value1, key2 => [value2, value3, ..], ...} # means to query: key1 == value1 && (key2 in [value2, value3, ..]) - def multi_field_query(items, size: 10_000) + def multi_field_query(items, size: 10_000, desc_keyword: nil) unless items warn 'empty filter!' exit @@ -31,6 +31,7 @@ class ESQuery } }, size: size } + query.merge!(assign_desc_body(desc_keyword)) if desc_keyword @client.search index: @index + '*', body: query end @@ -112,3 +113,11 @@ def parse_conditions(items) end items_hash end + +def assign_desc_body(keyword) + { + sort: [{ + keyword => { order: 'desc' } + }] + } +end -- 2.23.0

1 0

[PATCH v3 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..801649cf4 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || return + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

1 0

[PATCH compass-ci] 3-code/git: add repository version pointing
by Hu Xuejiao 28 Dec '20

28 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 2159ddc..48741ec 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -15,7 +15,7 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/compass-ci.git cd /c/compass-ci || exit - [ -n "$compass_commitid" ] && git reset --hard "$compass_commitid" + [ -n "$compass_commit_id" ] && git reset --hard "$compass_commit_id" ln -s compass-ci /c/cci } @@ -24,9 +24,12 @@ cd /c || exit git clone https://gitee.com/wu_fengguang/lkp-tests.git cd /c/lkp-tests || exit - [ -n "$lkp_commitid" ] && git reset --hard "$lkp_commitid" + [ -n "$lkp_commit_id" ] && git reset --hard "$lkp_commit_id" } [ -d "upstream-repos/.git" ] || { git clone https://gitee.com/wu_fengguang/upstream-repos.git + + cd /c/upstream-repos || exit + [ -n "$upstream_commit_id" ] && git reset --hard "$upstream_commit_id" } -- 2.23.0

1 0

[PATCH compass-ci] os/debian: add apt-get update
by Hu Xuejiao 28 Dec '20

28 Dec '20

[why] the url for downloading the package is displayed as 404 Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/0-package/os/debian | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/0-package/os/debian b/sparrow/0-package/os/debian index 6a58a0c..c45e8c2 100755 --- a/sparrow/0-package/os/debian +++ b/sparrow/0-package/os/debian @@ -22,4 +22,5 @@ pkgs=( hostname ) +apt-get update apt-get install -y --fix-missing "${pkgs[@]}" -- 2.23.0

1 0

[PATCH v3 lkp-tests] daemon: fix ssh login testbox for docker
by Xiao Shenwei 28 Dec '20

28 Dec '20

[why] systemctl is not supported when use docker Signed-off-by: Xiao Shenwei <xiaoshenwei96(a)163.com> --- daemon/sshd | 19 +++++++++++++++++-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/daemon/sshd b/daemon/sshd index 41fa99adf..c2404dbfd 100755 --- a/daemon/sshd +++ b/daemon/sshd @@ -5,20 +5,35 @@ # sshr_port_len # sshr_port_base -. $LKP_SRC/lib/http.sh +SCRIPT_DIR=$(dirname $(realpath $0)) +PROJECT_DIR=$(dirname $SCRIPT_DIR) + +. $PROJECT_DIR/lib/http.sh +. $PROJECT_DIR/lib/env.sh sshr_ip=$LKP_SERVER [ -n "$sshr_port" ] || sshr_port=5050 [ -n "$sshr_port_len" ] || sshr_port_len=2000 [ -n "$sshr_port_base" ] || sshr_port_base=50000 +generate_key_file() +{ + echo -e "y\n" | ssh-keygen -t rsa -b 2048 -N '' -f /etc/ssh/ssh_host_rsa_key +} + run_ssh() { [ -n "$my_ssh_pubkey" ] || return umask 0077 mkdir -p /root/.ssh echo "$my_ssh_pubkey" > /root/.ssh/authorized_keys - systemctl start sshd + if is_docker; then + generate_key_file + sshd_path=$(cmd_path sshd) + $sshd_path -q + else + systemctl start sshd + fi } data_success() -- 2.23.0

1 0

[PATCH compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 1

[PATCH compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 1

[PATCH compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..4544835 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load(File.open(File.join('/srv', "#{result_root}", 'job.yaml'))) +end -- 2.23.0

1 1

[PATCH v1 compass-ci 4/4] container: the start script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/start | 45 +++++++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/assist-result/start diff --git a/container/assist-result/start b/container/assist-result/start new file mode 100755 index 0000000..af812aa --- /dev/null +++ b/container/assist-result/start @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + ES_HOST + ES_PORT + ASSIST_RESULT_HOST + ASSIST_RESULT_PORT +] + +defaults = relevant_defaults(names) +env = docker_env(defaults) + +DEFAULT_LKP = '/c/lkp-tests' +DEFAULT_CCI = '/c/compass-ci' +DEFAULT_CONFIG_DIR = '/etc/compass-ci/defaults' +ASSIST_RESULT_PORT = defaults['ASSIST_RESULT_PORT'] +docker_rm 'assist-result' + +cmd = %w[ + docker run + --name assist-result + --restart=always + -d +] + env + %W[ + -e LKP_SRC=#{DEFAULT_LKP} + -e CCI_SRC=#{DEFAULT_CCI} + -p #{ASSIST_RESULT_PORT}:#{ASSIST_RESULT_PORT} + -v #{ENV['LKP_SRC']}:#{DEFAULT_LKP} + -v #{ENV['CCI_SRC']}:#{DEFAULT_CCI} + -v #{DEFAULT_CONFIG_DIR}:#{DEFAULT_CONFIG_DIR}:ro + -v /etc/localtime:/etc/localtime:ro + -v /srv/result:/srv/result:ro + -w #{DEFAULT_CCI}/container/assist-result/ + assist-result +] + +cmd += ['sh', '-c', 'umask 002 && ruby ./routes.rb'] + +system(*cmd) -- 2.23.0

1 0

[PATCH v1 compass-ci 3/4] container: the build script for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/build | 5 +++++ 1 file changed, 5 insertions(+) create mode 100755 container/assist-result/build diff --git a/container/assist-result/build b/container/assist-result/build new file mode 100755 index 0000000..130127b --- /dev/null +++ b/container/assist-result/build @@ -0,0 +1,5 @@ +#!/bin/sh +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +docker build -t assist-result . -- 2.23.0

1 0

[PATCH v1 compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..5672652 --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 0

[PATCH v1 compass-ci 1/4] container: add get_job_yaml in assist-result container
by Cao Xueliang 28 Dec '20

28 Dec '20

The assist-result container mount the "/srv/result" dir, we should implement some API about "/srv/result". An API is provided to support to get job yaml for now. Example: input: curl localhost:8102/get_job_yaml/crystal.304189 output: the yaml content about crystal.304189 Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 24 +++++++++++++++++++ container/assist-result/views/get_job_yaml.rb | 13 ++++++++++ 2 files changed, 37 insertions(+) create mode 100755 container/assist-result/routes.rb create mode 100644 container/assist-result/views/get_job_yaml.rb diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb new file mode 100755 index 0000000..1013643 --- /dev/null +++ b/container/assist-result/routes.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'sinatra' + +require_relative './views/get_job_yaml' + +configure do + set :bind, '0.0.0.0' + set :port, ENV['ASSIST_RESULT_PORT'] +end + +get '/get_job_yaml/:job_id' do + begin + result = get_job_yaml(params[:job_id]) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assist-result/views/get_job_yaml.rb b/container/assist-result/views/get_job_yaml.rb new file mode 100644 index 0000000..4544835 --- /dev/null +++ b/container/assist-result/views/get_job_yaml.rb @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'yaml' + +require_relative "#{ENV['CCI_SRC']}/lib/es_query" + +def get_job_yaml(job_id) + content = ESQuery.new.query_by_id(job_id) + result_root = content['result_root'] + return YAML.load(File.open(File.join('/srv', "#{result_root}", 'job.yaml'))) +end -- 2.23.0

1 0

[PATCH compass-ci 2/4] container: the Dockerfile for assist-result
by Cao Xueliang 28 Dec '20

28 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/Dockerfile | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 container/assist-result/Dockerfile diff --git a/container/assist-result/Dockerfile b/container/assist-result/Dockerfile new file mode 100644 index 0000000..5672652 --- /dev/null +++ b/container/assist-result/Dockerfile @@ -0,0 +1,13 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. + +FROM alpine:3.11 + +RUN sed -ri.origin 's|^https?://dl-cdn.alpinelinux.org|http://mirrors.huaweicloud.com|g' /etc/apk/repositories && \ + apk update && \ + apk add --no-cache 'ruby-dev' 'ruby-etc' \ + 'g++' 'gcc' 'pcre' 'libevent' 'make' 'git' 'bash' 'grep' 'coreutils' 'curl' 'util-linux' + +RUN umask 002 && \ + echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ + gem install rest-client activesupport json yaml elasticsearch sinatra puma -- 2.23.0

1 1

[PATCH compass-ci] lib/params_group.rb optimize filter_groups
by Lu Weitao 28 Dec '20

28 Dec '20

[Why] We had for-each in hash, needn't query value by key once more Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/params_group.rb | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 7bb7f5b..bf229e6 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -72,10 +72,10 @@ end def filter_groups(groups) groups.each do |group_key, value| - value.each_key do |dim_key| - value.delete(dim_key) if value[dim_key].empty? + value.each do |dim_key, job_list| + value.delete(dim_key) if job_list.empty? end - groups.delete(group_key) if groups[group_key].empty? + groups.delete(group_key) if value.empty? end end -- 2.23.0

2 5

[PATCH v2 lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[error] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..29b423b69 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,7 +48,10 @@ get_config() upload_config() { - upload_to_target_dir "${cci_dir}/${pkgname}" + local upload_dir="${cci_dir}/${pkgname}" + [ -d "${upload_dir}" ] || mkdir -p "${upload_dir}" + + upload_to_target_dir "${upload_dir}" } mount_dest() -- 2.23.0

1 0

[PATCH compass-ci] lib/json_logger: delete caller
by Wu Zhende 28 Dec '20

28 Dec '20

caller does not provide error information. It's not helpful for fault locating. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- lib/json_logger.rb | 1 - src/lib/json_logger.cr | 1 - 2 files changed, 2 deletions(-) diff --git a/lib/json_logger.rb b/lib/json_logger.rb index 6d1173e..78f95ca 100644 --- a/lib/json_logger.rb +++ b/lib/json_logger.rb @@ -25,7 +25,6 @@ class JSONLogger < Logger } logger_hash['progname'] = progname if progname - logger_hash['caller'] = caller if level_num >= 2 msg = { 'message' => msg } unless msg.is_a?(Hash) logger_hash.merge!(msg) diff --git a/src/lib/json_logger.cr b/src/lib/json_logger.cr index 6d61197..fcd65fb 100644 --- a/src/lib/json_logger.cr +++ b/src/lib/json_logger.cr @@ -25,7 +25,6 @@ class JSONLogger < Logger })).as_h logger_hash.any_merge!({"progname" => progname}) unless progname.empty? - logger_hash.merge!(JSON.parse(%({"caller": #{caller}})).as_h) if level_num >= 2 begin message = JSON.parse(msg).as_h -- 2.23.0

1 0

[PATCH lkp-tests] tests/build-pkg: fix directory cannot access
by Liu Shaofei 28 Dec '20

28 Dec '20

[why] upload_config upload_to_target_dir /cci/build-config/vue . /lkp/lkp/src/lib/upload.sh . /lkp/lkp/src/lib/env.sh '[' -n 1 ']' return local dir=/cci/build-config/vue grep -v @ ls -F /cci/build-config/vue ls: cannot access '/cci/build-config/vue': No such file or directory Signed-off-by: Liu Shaofei <370072077(a)qq.com> --- tests/build-pkg | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index 44bd1c10d..f6c27a754 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -48,6 +48,8 @@ get_config() upload_config() { + [ -d "${cci_dir}/${pkgname}" ] || mkdir -p "${cci_dir}/${pkgname}" + upload_to_target_dir "${cci_dir}/${pkgname}" } -- 2.23.0

2 2

[PATCH compass-ci] 3-code/git: iadd repository version pointing
by Hu Xuejiao 28 Dec '20

28 Dec '20

[why] the code downloaded from upstream_repo occupies too much memory, the repository version needs to specify a version Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/3-code/git | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/3-code/git b/sparrow/3-code/git index 2159ddc..e2b3cbf 100755 --- a/sparrow/3-code/git +++ b/sparrow/3-code/git @@ -29,4 +29,7 @@ cd /c || exit [ -d "upstream-repos/.git" ] || { git clone https://gitee.com/wu_fengguang/upstream-repos.git + + cd /c/upstream-repos || exit + [ -n "$upstream_commitid" ] && git reset --hard "$upstream_commitid" } -- 2.23.0

2 1

[PATCH compass-ci 5/6] mail-robot: add error code/message
by Luan Shengde 28 Dec '20

28 Dec '20

add error message for each error code before building the email_message. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- .../lib/assign-account-fail-eamil.rb | 87 +++++++++++++++---- 1 file changed, 69 insertions(+), 18 deletions(-) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 593f345..5871a6a 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -3,32 +3,83 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. # frozen_string_literal: true -def build_apply_account_fail_email(my_info, message) - email_msg = <<~EMAIL_MESSAGE - To: #{my_info['my_email']} - Subject: [compass-ci] apply account failed +def email_err_message(message) + case message + when 'NO_COMMIT_URL' + err_message = <<~EMAIL_MESSAGE + No commit url found. + You should add a commit url for the 'apply account' email. - Dear user: + For example: + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'URL_PREFIX_ERR' + err_message = <<~EMAIL_MESSAGE + Please add a correct prefix for the commit url. + + my oss commit + + For example: + + my oss commit: https://github.com/torvalds/linux/commit/7be74942f184fdfba34ddd19a0d995deb3… + EMAIL_MESSAGE + when 'NOT_REGISTERED' + err_message = <<~EMAIL_MESSAGE + Your repo has not been registered to our upstream-repos yet. + You should register your repo to the upstream first. + Try again after you have done it. + + Reference the following url to learn how to register the repo: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/test-oss-p… + EMAIL_MESSAGE + when 'COMMIT_URL_NOT_AVAILABLE' + err_message = <<~EMAIL_MESSAGE + We cannot confirm the commit url matches your email. + + Make sure the commit exists. + Make sure it is truely submitted with you email. - Your application for account failed with following error: + Go to the commit history and find your commit. + Enter the commit and you will get your commit url. - #{message} + Try again after you have correctted the url + EMAIL_MESSAGE + when 'NO_PUBKEY' + err_message = <<~EMAIL_MESSAGE + No pub_key found. + Please add a pubkey to the email and then try again. + EMAIL_MESSAGE + when 'PUBKEY_NAME_ERR' + err_message = <<~EMAIL_MESSAGE + Pubkey file name error, keep it's name as when it was generated. + The pubkey file name should like: - In order to successfully apply an account, please pay attention to the following points: + id_{{ xxx }}.pub + EMAIL_MESSAGE + end - 1. mail subject - The subject should exactly: apply account + err_message += <<~EMAIL_MESSAGE - 2. commit url - When you writing the url, add prefix: my oss commit - example: - my oss commit: https://github.com/torvalds/aalinux/commit/7be74942f184fdfba34ddd19a0d995de… + Manual for how to apply account: + + https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manual/apply-acco… + EMAIL_MESSAGE + + err_message +end + +def build_apply_account_fail_email(my_info, message) + err_message = email_err_message(message) + email_msg = <<~EMAIL_MESSAGE + To: #{my_info['my_email']} + Subject: [compass-ci] apply account failed + + Dear user: - attention: - Ensure you commit url exist and available to access. + Your application for account failed. - 3. ssh pubkey - You need to add a pubkey as an attachment to the email. + #{err_message} regards compass-ci -- 2.23.0

1 0

[PATCH compass-ci 4/6] mail-robot: optimize message message
by Luan Shengde 28 Dec '20

28 Dec '20

optimize mail message for successfully applied account [why] its used linux command 'curl' to send request data when sending email. so the ${HOME} and ${SHELL##*/} is treated as a system variable and replaced with the variable value. as a result, we will send a unexpected email. to avoid this problem, need add '\\' before '$' in the old. [how] use rest-client instead of command 'curl' to send the request. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-email.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/mail-robot/lib/assign-account-email.rb b/container/mail-robot/lib/assign-account-email.rb index 466586d..ee3b574 100755 --- a/container/mail-robot/lib/assign-account-email.rb +++ b/container/mail-robot/lib/assign-account-email.rb @@ -37,7 +37,7 @@ def build_apply_account_email(my_info) git clone https://gitee.com/wu_fengguang/lkp-tests.git cd lkp-tests make install - source \\${HOME}/.\\${SHELL##*/}rc + source ${HOME}/.${SHELL##*/}rc 3) submit job reference: https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/tutorial.md -- 2.23.0

1 0

[PATCH compass-ci 3/6] mail-robot: optimize apply-jumper-account
by Luan Shengde 28 Dec '20

28 Dec '20

use rest-client instead of linux command 'curl' to send http request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-jumper-account.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/mail-robot/lib/apply-jumper-account.rb b/container/mail-robot/lib/apply-jumper-account.rb index 44f412b..73d2946 100755 --- a/container/mail-robot/lib/apply-jumper-account.rb +++ b/container/mail-robot/lib/apply-jumper-account.rb @@ -22,8 +22,8 @@ class ApplyJumperAccount end def apply_jumper_account - account_info_str = %x(curl -XGET "#{@jumper_host}:#{@jumper_port}/assign_account" \ - -d '#{(a)my_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, @my_info.to_json account_info = JSON.parse account_info_str account_info_exist(account_info) -- 2.23.0

1 0

[PATCH compass-ci 2/6] mail-robot: optimize apply-account
by Luan Shengde 28 Dec '20

28 Dec '20

use rest-client instead of linux command 'curl' to send request Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/apply-account.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/lib/apply-account.rb b/container/mail-robot/lib/apply-account.rb index aea1aa5..e8288d7 100755 --- a/container/mail-robot/lib/apply-account.rb +++ b/container/mail-robot/lib/apply-account.rb @@ -5,6 +5,7 @@ require 'json' require 'mail' +require 'rest-client' require_relative '../../../lib/es_client' require_relative 'assign-account-email' require_relative 'assign-account-fail-eamil' @@ -131,6 +132,7 @@ class ApplyAccount build_apply_account_fail_email(@my_info, error_message) end - %x(curl -XPOST "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" -d "#{email_message}") + send_mail_url = "#{@send_mail_host}:#{@send_mail_port}/send_mail_text" + RestClient.post send_mail_url, email_message end end -- 2.23.0

1 0

[PATCH compass-ci 1/6] mail-robot: update Dockerfile
by Luan Shengde 28 Dec '20

28 Dec '20

[why] the old way is use linux command 'curl' to send http request. it's not human friendly to read for user [how] install rest-client instead of linux command 'curl' Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/Dockerfile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/container/mail-robot/Dockerfile b/container/mail-robot/Dockerfile index 76dc94d..f0aebbd 100644 --- a/container/mail-robot/Dockerfile +++ b/container/mail-robot/Dockerfile @@ -8,7 +8,8 @@ MAINTAINER Luan Shengde <shdluan(a)163.com> ENV DEBIAN_FRONTEND noninteractive RUN apt-get update && \ - apt-get install -y git uuid-runtime curl ruby-listen ruby-json ruby-mail + apt-get install -y git uuid-runtime ruby-listen ruby-json ruby-mail ruby-rest-client + RUN echo ':sources: ["http://rubygems.org"]' >> ~/.gemrc && \ gem install fileutils elasticsearch activesupport -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: fix spell error
by Lu Kaiyi 28 Dec '20

28 Dec '20

Signed-off-by: Lu Kaiyi <2392863668(a)qq.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 0ca71e1..ea67064 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -328,7 +328,7 @@ def get_banner(git_repo, branches) } end -def get_optimize_jobs_braches(jobs) +def get_optimize_jobs_branches(jobs) branch_set = Set.new jobs.size.times do |i| branch = jobs[i]['upstream_branch'].to_s @@ -347,7 +347,7 @@ def get_jobs_body(params) page_size = get_positive_number(params.delete(:page_size), 20) page_num = get_positive_number(params.delete(:page_num), 1) - 1 jobs, total = search_job(params, page_size, page_num) - jobs, branches = get_optimize_jobs_braches(jobs) + jobs, branches = get_optimize_jobs_branches(jobs) { total: total, filter: params, -- 2.23.0

1 0

[PATCH compass-ci] 1-storage/tiny: add mount path
by Hu Xuejiao 26 Dec '20

26 Dec '20

Signed-off-by: Hu XueJiao <1034502035(a)qq.com> --- sparrow/1-storage/tiny | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index e589937..eb7270d 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,3 +26,6 @@ dirs=( ) mkdir -p "${dirs[@]}" + +mount -t cifs //$LKP_SERVER/os /srv/os -o guest,ro,hard,vers=1.0,noacl,nouser_xattr +mount -t cifs //$LKP_SERVER/initrd /srv/initrd -o port=446,guest,ro,hard,vers=1.0,noacl,nouser_xattr -- 2.23.0

1 0

[PATCH compass-ci 3/4] assign-account: optimize answerback-email
by Luan Shengde 26 Dec '20

26 Dec '20

[why] the old way to send request is use linux command 'curl', it's not good for reading and has low efficiency use ruby gem resource rest-client instead of linux command Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 21285e7..a45fef2 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -11,6 +11,7 @@ require 'json' require 'mail' require 'set' require 'optparse' +require 'rest-client' require_relative '../defconfig' require_relative '../../lib/es_client' require_relative 'build-send-account-email' @@ -180,7 +181,8 @@ def apply_account(my_info, conf_info) apply_info.update conf_info apply_info['lab'] = LAB - account_info_str = %x(curl -XGET '#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account' -d '#{apply_info.to_json}') + assign_account_url = "#{JUMPER_HOST}:#{JUMPER_PORT}/assign_account" + account_info_str = RestClient.post assign_account_url, apply_info.to_json JSON.parse account_info_str end @@ -285,7 +287,8 @@ def send_mail(my_info, account_info, conf_info) build_message(my_info['my_email'], account_info) end - %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") + send_mail_url = "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" + RestClient.post send_mail_url, message end def store_account_info(my_info) -- 2.23.0

1 0