April 2021 - Compass-ci - mailweb.openeuler.org

[PATCH v2 lkp-tests 2/3] install-iso: fix some bugs
by Yu Chuan 27 Apr '21

27 Apr '21

[What Bugs] 1. optimize name: reboot.cgz => reboot-after-systemd-logind.cgz 2. nfs mount result dir: - before: ${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} - after : ${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} 3. exit directory before umount Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 6 +++--- tests/install-iso-sync-rootfs | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c51b3784b5aa..481b99e35df4 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -196,8 +196,8 @@ submit_install_iso_firstboot_yaml() custom_ipxe="#!ipxe\\\\n" custom_ipxe="${custom_ipxe}\\\\ndhcp" custom_ipxe="${custom_ipxe}\\\\ninitrd http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/initrd.lkp" - custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot.cgz" - custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot.cgz local rw" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot-after-systemd-logind.cgz" + custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot-after-systemd-logind.cgz local rw" custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" $firstboot_yaml @@ -225,7 +225,7 @@ submit_install_iso_sync_rootfs_yaml() install-iso-sync-rootfs.NFS_ROOT_DIR=${NFS_ROOT_DIR} \ install-iso-sync-rootfs.ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ install-iso-sync-rootfs.VG_NAME=${VG_NAME} \ - testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" + testbox=$testbox || die "submit job failed: ${sync_rootfs_yaml}" } main() diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index 34096229ccfb..b74c7bed2044 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -59,7 +59,7 @@ sync_rootfs_to_nfs_server() local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} mkdir -p ${nfs_rootfs_mount_point} - mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" # activate the lvm vgchange -ay @@ -78,7 +78,7 @@ sync_rootfs_to_nfs_server() cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" # umount nfs rootfs - umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + cd / && umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" # umount local logical volume umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] install-iso-pre: change the logic of ks generate and upload
by Yu Chuan 27 Apr '21

27 Apr '21

[Why] 1. 'key difference' of ks for different physical machine: - Choice Install Source - Choice Install Partition - Choice working Nic 2. 'other ks content' for different os is different. so the change is as following: before: 'other ks content' is all same. after : add a httpd dir, which stores the 'other ks content' for different os, then they only need generate the 'key difference' related to current physical machine, 'other ks content' can download from httpd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 36 ++---------------------------------- 1 file changed, 2 insertions(+), 34 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c89895a994cc..c51b3784b5aa 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -112,41 +112,9 @@ generate_ks() bootloader --location=none logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting - # Root password - rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} - - # Reboot after installation - reboot --eject - - # Use text mode install - text - - # Keyboard layouts - keyboard --vckeymap=us --xlayouts='us' - - # System language - lang en_US.UTF-8 - - # Do not configure the X Window System - skipx - - # System services - services --enabled="chronyd" - - # System timezone - timezone Asia/Shanghai --isUtc - - %packages - @^minimal-environment - - %end - - %anaconda - pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty - pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok - pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty - %end EOF + + curl -s http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/install/ks/template/${iso_os}-${iso_arch}-${iso_version}.ks >> $ks_file } upload_ks() -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: use assistant client send account mail
by Cao Xueliang 27 Apr '21

27 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 38828a3..1112f3b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -84,12 +84,7 @@ class MailBisectResult 'my_name' => @git_commit.author_name, 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } - - account_info = ESQuery.new(index: 'accounts').query_by_id(@to) - return if account_info - - apply_account = AutoAssignAccount.new(user_info) - apply_account.send_account + AssistantClient.new.send_account_mail(user_info) end def rm_work_dir -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: rename $job_state
by Liu Yinsi 27 Apr '21

27 Apr '21

job_state => report_content See [PATCH lkp-tests 1/3] tests/self-test: send mail when failed to submit job Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index 24ef7bb64..b5e56df79 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -89,7 +89,7 @@ Dear $my_name: Your self-test groud_id=selftest job result is: - $job_state + $report_content Regards Compass-CI -- 2.23.0

1 0

[PATCH compass-ci] container/assistant: add send_account_mail function
by Cao Xueliang 27 Apr '21

27 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 12 +++++++++++ .../assistant/views/send_account_mail.rb | 20 +++++++++++++++++++ lib/assign_account_client.rb | 13 +++++++----- lib/assistant_client.rb | 8 ++++++++ 4 files changed, 48 insertions(+), 5 deletions(-) create mode 100644 container/assistant/views/send_account_mail.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb index 232ec8c..9a96fc0 100755 --- a/container/assistant/routes.rb +++ b/container/assistant/routes.rb @@ -9,6 +9,7 @@ require 'open3' require_relative './views/locate_files' require_relative './views/get_mail_list' +require_relative './views/send_account_mail' configure do set :bind, '0.0.0.0' @@ -37,3 +38,14 @@ get '/get_mail_list/:type' do return [200, result.to_json] end + +post '/send_account_mail' do + begin + data = JSON.parse(request.body.read) + result = send_account_mail(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assistant/views/send_account_mail.rb b/container/assistant/views/send_account_mail.rb new file mode 100644 index 0000000..b96d082 --- /dev/null +++ b/container/assistant/views/send_account_mail.rb @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# # frozen_string_literal: true +# +require_relative "#{ENV['CCI_SRC']}/lib/es_query" +require_relative "#{ENV['CCI_SRC']}/lib/assign_account_client" + +def send_account_mail(user_info) + account_info = ESQuery.new(index: 'accounts').query_by_id(user_info['my_email']) + return {'status'=> true} if account_info + + begin + apply_account = AutoAssignAccount.new(user_info) + apply_account.send_account + rescue + return {'status' => false} + end + + return {'status' => true} +end diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index f551a9d..5aad955 100644 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -4,13 +4,13 @@ # frozen_string_literal: true require 'json' -require 'mail' require 'set' require 'optparse' require 'rest-client' require_relative '../container/defconfig' require_relative 'es_client' require_relative '../container/mail-robot/lib/assign-account-email' +require_relative "#{ENV['CCI_SRC']}/lib/parse_mail_list" names = Set.new %w[ JUMPER_HOST @@ -32,6 +32,7 @@ class AutoAssignAccount @my_info = user_info @my_info_es = {} + @account_info = {} end def update_from_es @@ -59,10 +60,10 @@ class AutoAssignAccount end def update_my_info_from_account_info - account_info = apply_account + @account_info = apply_account - @my_info['my_login_name'] = account_info['my_login_name'] - @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + @my_info['my_login_name'] = @account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << @account_info['my_jumper_pubkey'] unless @account_info['my_jumper_pubkey'].nil? end def store_account_info @@ -72,7 +73,9 @@ class AutoAssignAccount def send_mail @my_info['bisect'] = true - message = build_apply_account_email(@my_info) + default_email = parse_mail_list('delimiter') + @my_info['my_email'] = default_email['to'] unless default_email.empty? + message = build_apply_account_email(@my_info, @account_info, false) %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") end diff --git a/lib/assistant_client.rb b/lib/assistant_client.rb index c951f2b..6774e9e 100644 --- a/lib/assistant_client.rb +++ b/lib/assistant_client.rb @@ -21,4 +21,12 @@ class AssistantClient return JSON.parse(response.body) end + + def send_account_mail(user_info) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_account_mail") + response = resource.post(user_info.to_json) + return nil unless response.code == 200 + + return JSON.parse(response.body) + end end -- 2.23.0

1 0

[PATCH lkp-tests 3/3] jobs: remove multi-docker/qemu
by Liu Yinsi 27 Apr '21

27 Apr '21

multi-qemu and multi-docker will deploy a simply ci env by calling compass-ci/sparrow/install-client, but deploy-cci will deploy a ci env by calling compass-ci/sparrow/install-tiny, they are 2 defferent envs, so remove multi-docker/qemu. testboxes for docker and qemu will be ready by sparrow/8-testbox, 8-testbox already be called by sparrow/install-tiny. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- jobs/self-test.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/jobs/self-test.yaml b/jobs/self-test.yaml index ad2dcd2a2..b0c1c08c3 100644 --- a/jobs/self-test.yaml +++ b/jobs/self-test.yaml @@ -2,5 +2,4 @@ suite: self-test category: functional deploy-cci: -multi-qemu: -multi-docker: +self-test: -- 2.23.0

1 0

[PATCH lkp-tests 2/3] tests/self-test: fix command not found
by Liu Yinsi 27 Apr '21

27 Apr '21

[error] /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 24: es-jobs: command not found Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/self-test | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/self-test b/tests/self-test index 6e4ca7ef2..fb3e87b17 100755 --- a/tests/self-test +++ b/tests/self-test @@ -3,6 +3,8 @@ . $LKP_SRC/lib/email.sh . $LKP_SRC/lib/log.sh +source /etc/profile.d/compass.sh + submit_args=( "testbox=dc-8g cci-depends.benchmark=sshd cci-depends.yaml" "testbox=dc-8g borrow-1h.yaml runtime=1m" -- 2.23.0

1 0

[PATCH lkp-tests 1/3] tests/self-test: send mail when failed to submit job
by Liu Yinsi 27 Apr '21

27 Apr '21

When failed to deploy ci env, some services not ready like es, sched, can't query anything from ES, and email content $job_state is null, so send email when failed to submit job. function submit_one() is redundant, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/self-test | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/tests/self-test b/tests/self-test index cb06b7226..6e4ca7ef2 100755 --- a/tests/self-test +++ b/tests/self-test @@ -1,6 +1,7 @@ #!/bin/bash . $LKP_SRC/lib/email.sh +. $LKP_SRC/lib/log.sh submit_args=( "testbox=dc-8g cci-depends.benchmark=sshd cci-depends.yaml" @@ -8,21 +9,22 @@ submit_args=( "testbox=vm-2p8g borrow-1h.yaml runtime=1m" ) -submit_one() -{ - args="$1" - submit_result=$(submit -m $args group_id=selftest) -} - -submit_all() +submit_job() { for args in "${submit_args[@]}" do - submit_one "$args" + report_content=$(submit -m $args group_id=selftest 2>&1) + echo $report_content | grep -q "\"job_state\":\"submit\"" || + { + log_error "submit $args failed: $report_content" + send_email selftest + exit 1 + } done - job_state=$(es-jobs group_id=selftest | grep job_state) + report_content=$(es-jobs group_id=selftest | grep job_state) + log_info "selftest job result: $report_content" } -submit_all +submit_job send_email selftest -- 2.23.0

1 0

[PATCH lkp-tests] iso2rootfs: some optimize
by Yu Chuan 27 Apr '21

27 Apr '21

1. change dailybuild result rootfs dir name why: - make the dailybuild rootfs directory more meaningful. how: - before: /srv/os/${iso_os}/${iso_arch}/${iso_version}-${TIMESTAMP} - after : /srv/os/${iso_os}/${iso_arch}/${iso_version}-dailybuild-${TIMESTAMP} 2. change sha256sum file why: - unify the file name, then easier to statistics. how: - file name: - before: ${ROOTFS_DES_DIR}/${iso_version}-latest.sha256sum - after : ${ROOTFS_DES_DIR}/iso.sha256sum - file content: - before: 168d2a8c24d20acdf06e67ffee90174b321b52e36afbddfc41602454feb2c09c openEuler-20.03-LTS-aarch64-dvd.iso - after : 168d2a8c24d20acdf06e67ffee90174b321b52e36afbddfc41602454feb2c09c Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 06c8565110ab..e8694a73e807 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -318,6 +318,8 @@ download_compass_ci() config_rootfs_dir() { ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") + [ -n "${dailybuild_iso_url_file}" ] && ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-dailybuild-$(date "+%Y-%m-%d-%H-%M-%S") + [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -592,7 +594,7 @@ update_dailybuild_soft_link() [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}" + echo $SHA256SUM_NET > "${ROOTFS_DES_DIR}/iso.sha256sum" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && -- 2.23.0

2 2

[PATCH lkp-tests 2/3] install-iso-pre: change the logic of ks generate and upload
by Yu Chuan 27 Apr '21

27 Apr '21

[Why] 1. 'key difference' of ks for different physical machine: - http install source. - install target disk/partition. - working nic. 2. 'other ks content' for different os is different. so the change is as following: before: 'other ks content' is all same. after : add a httpd dir, which stores the 'other ks content' for different os, then they only need generate the 'key difference' related to current physical machine, 'other ks content' can download from httpd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 43 +++++++++---------------------------------- 1 file changed, 9 insertions(+), 34 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c89895a994cc..f9ff9ebe9e94 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -91,6 +91,13 @@ prepare_lv() create_lv } +append_ks_template() +{ + wget -O ${ks_file}.tmp http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/install/ks/template/${iso_os}-${iso_arch}-${iso_version}.ks + cat ${ks_file}.tmp >> $ks_file + rm -rf ${ks_file}.tmp +} + generate_ks() { log_info "start generate $ks_file" @@ -112,41 +119,9 @@ generate_ks() bootloader --location=none logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting - # Root password - rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} - - # Reboot after installation - reboot --eject - - # Use text mode install - text - - # Keyboard layouts - keyboard --vckeymap=us --xlayouts='us' - - # System language - lang en_US.UTF-8 - - # Do not configure the X Window System - skipx - - # System services - services --enabled="chronyd" - - # System timezone - timezone Asia/Shanghai --isUtc - - %packages - @^minimal-environment - - %end - - %anaconda - pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty - pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok - pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty - %end EOF + + append_ks_template } upload_ks() -- 2.23.0

2 2