- Compass-ci - mailweb.openeuler.org

[PATCH v6 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 28 Apr '21

28 Apr '21

To make sure git_repos displayed on the web are right. examples of git_repo: a/AvxToNeon/AvxToNeon 0/caterqiu/001 aur-a/a2png/a2png m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 9e899d9..e305ac6 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -400,6 +400,14 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + + # What does this regular expression want: + # 1. a three-segment structure "xxx/xxx/xxx" + # 2. there can be '-' or '_' in every segment, but can't be at first or last of the segment. + # 3. the first segment can have lowercase of letters or numbers in it. + # 4. the other two segments can have letters(lowercase or uppercase) or numbers in it. + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-zA-Z0-9][\w\-]*[a-zA-Z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH lkp-tests] jobs/send-email.yaml: change the default value.
by Li Ping 28 Apr '21

28 Apr '21

the default value of email subject should not be job_failed, and there is already a send-email-on-fail.yaml. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/send-email.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml index 8f8b48b96..2689634c2 100644 --- a/jobs/send-email.yaml +++ b/jobs/send-email.yaml @@ -3,4 +3,4 @@ # values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. send_email: - subject: job_failed + subject: -- 2.23.0

1 0

[PATCH compass-ci 2/2] kibana/dashboard_import: import kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can import kibana dashboards by json files. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_import | 45 +++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/kibana/dashboard_import diff --git a/container/kibana/dashboard_import b/container/kibana/dashboard_import new file mode 100755 index 0000000..670fa3f --- /dev/null +++ b/container/kibana/dashboard_import @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'pathname' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +files = [] +ARGV.each do |f| + files >> f +end + +if files.empty? + profile = Pathname.new(File.dirname(__FILE__)).realpath + files << profile + 'data.json' + files << profile + 'resource.json' + files << profile + 'all.json' +end +puts files + +cmd = "curl -XPOST -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/import" +cmd += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + +files.each do |f| + new_cmd = cmd + new_cmd += " -d @#{f}" + puts new_cmd + system new_cmd +end -- 2.23.0

2 1

[PATCH compass-ci] src/lib/web_backend.rb: modify parameter list for web compare
by Li Yuanchao 28 Apr '21

28 Apr '21

The parameter list for web compare need to change, as tbox_groups have $user at the back. Suite and tbox_group whose count < 2 need not to show, cause have no one to compare. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index da3c3c5..a9ab091 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -45,16 +45,11 @@ def es_count(query) ES_CLIENT.count(index: 'jobs*', body: query)['count'] end -# "vm-2p8g--212" remove "--212" -# "vm-2p8g--zzz" remove "--zzz" -# "vm-git-bisect" don't remove "-bisect" +# delete $user after '--' or '.' or '~' def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ - index = r.index('--') || r.rindex('-') - r = r[0, index] - end + r = r.gsub(/(--|\.|~).*$/, '') result.add r end result.to_a @@ -70,10 +65,10 @@ def all_suite size: 0 } es_result = es_query(query)['aggregations']['all_suite']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - - es_result + result_second = [] + es_result.each { |e| result_second << e if e['doc_count'] > 1 } + result_second.sort_by! { |h| h['doc_count'] } + result_second.reverse!.map! { |x| x['key'] } end def all_tbox_group @@ -86,10 +81,12 @@ def all_tbox_group size: 0 } es_result = es_query(query)['aggregations']['all_tbox_group']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } + result_second = [] + es_result.each { |e| result_second << e if e['doc_count'] > 1 } + result_second.sort_by! { |h| h['doc_count'] } + result_second.reverse!.map! { |x| x['key'] } - filter_tbox_group(es_result) + filter_tbox_group(result_second) end def compare_candidates_body -- 2.23.0

2 1

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: add field
by Lu Weitao 28 Apr '21

28 Apr '21

add nr_run(integer), author_email(keyword) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 67b1ed14..0c2fc62d 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -148,6 +148,12 @@ else "upstream_commit": { "type": "keyword" }, + "base_commit": { + "type": "keyword" + }, + "nr_run": { + "type": "integer" + }, "my_email": { "type": "keyword" }, -- 2.23.0

2 2

[PATCH compass-ci 1/2] kibana/dashboard_export: exporting kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can export kibana dashboards by ids. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_export | 37 +++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/kibana/dashboard_export diff --git a/container/kibana/dashboard_export b/container/kibana/dashboard_export new file mode 100755 index 0000000..d6e0054 --- /dev/null +++ b/container/kibana/dashboard_export @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +dashboard_ids = [] +ARGV.each do |i| + dashboard_ids << i +end + +dashboard_ids = %w[data resource all] if dashboard_ids.empty? + +cmd = "curl -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/export?dashboard=" + +dashboard_ids.each do |id| + new_cmd = cmd + new_cmd += "#{id} >> #{id}.json" + puts new_cmd + system new_cmd +end -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] iso2rootfs: some optimize
by Yu Chuan 27 Apr '21

27 Apr '21

1. change dailybuild result rootfs dir name why: - make the dailybuild rootfs directory more meaningful. how: - before: /srv/os/${iso_os}/${iso_arch}/${iso_version}-${TIMESTAMP} - after : /srv/os/${iso_os}/${iso_arch}/${iso_version}-dailybuild-${TIMESTAMP} 2. change sha256sum file why: - unify the file name, then easier to statistics. how: - file name: - before: ${ROOTFS_DES_DIR}/${iso_version}-latest.sha256sum - after : ${ROOTFS_DES_DIR}/SHA256SUM Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 06c8565110ab..ca972b2399f6 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -318,6 +318,8 @@ download_compass_ci() config_rootfs_dir() { ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") + [ -n "${dailybuild_iso_url_file}" ] && ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-dailybuild-$(date "+%Y-%m-%d-%H-%M-%S") + [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -592,7 +594,7 @@ update_dailybuild_soft_link() [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}" + cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}/SHA256SUM" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] install-iso: fix some bugs
by Yu Chuan 27 Apr '21

27 Apr '21

[What Bugs] 1. optimize name: reboot.cgz => reboot-after-systemd-logind.cgz 2. nfs mount result dir: - before: ${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} - after : ${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} 3. exit directory before umount Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 6 +++--- tests/install-iso-sync-rootfs | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c51b3784b5aa..481b99e35df4 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -196,8 +196,8 @@ submit_install_iso_firstboot_yaml() custom_ipxe="#!ipxe\\\\n" custom_ipxe="${custom_ipxe}\\\\ndhcp" custom_ipxe="${custom_ipxe}\\\\ninitrd http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/initrd.lkp" - custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot.cgz" - custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot.cgz local rw" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot-after-systemd-logind.cgz" + custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot-after-systemd-logind.cgz local rw" custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" $firstboot_yaml @@ -225,7 +225,7 @@ submit_install_iso_sync_rootfs_yaml() install-iso-sync-rootfs.NFS_ROOT_DIR=${NFS_ROOT_DIR} \ install-iso-sync-rootfs.ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ install-iso-sync-rootfs.VG_NAME=${VG_NAME} \ - testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" + testbox=$testbox || die "submit job failed: ${sync_rootfs_yaml}" } main() diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index 34096229ccfb..b74c7bed2044 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -59,7 +59,7 @@ sync_rootfs_to_nfs_server() local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} mkdir -p ${nfs_rootfs_mount_point} - mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" # activate the lvm vgchange -ay @@ -78,7 +78,7 @@ sync_rootfs_to_nfs_server() cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" # umount nfs rootfs - umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + cd / && umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" # umount local logical volume umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] install-iso-pre: change the logic of ks generate and upload
by Yu Chuan 27 Apr '21

27 Apr '21

[Why] 1. 'key difference' of ks for different physical machine: - Choice Install Source - Choice Install Partition - Choice working Nic 2. 'other ks content' for different os is different. so the change is as following: before: 'other ks content' is all same. after : add a httpd dir, which stores the 'other ks content' for different os, then they only need generate the 'key difference' related to current physical machine, 'other ks content' can download from httpd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 36 ++---------------------------------- 1 file changed, 2 insertions(+), 34 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c89895a994cc..c51b3784b5aa 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -112,41 +112,9 @@ generate_ks() bootloader --location=none logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting - # Root password - rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} - - # Reboot after installation - reboot --eject - - # Use text mode install - text - - # Keyboard layouts - keyboard --vckeymap=us --xlayouts='us' - - # System language - lang en_US.UTF-8 - - # Do not configure the X Window System - skipx - - # System services - services --enabled="chronyd" - - # System timezone - timezone Asia/Shanghai --isUtc - - %packages - @^minimal-environment - - %end - - %anaconda - pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty - pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok - pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty - %end EOF + + curl -s http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/install/ks/template/${iso_os}-${iso_arch}-${iso_version}.ks >> $ks_file } upload_ks() -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: use assistant client send account mail
by Cao Xueliang 27 Apr '21

27 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 38828a3..1112f3b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -84,12 +84,7 @@ class MailBisectResult 'my_name' => @git_commit.author_name, 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } - - account_info = ESQuery.new(index: 'accounts').query_by_id(@to) - return if account_info - - apply_account = AutoAssignAccount.new(user_info) - apply_account.send_account + AssistantClient.new.send_account_mail(user_info) end def rm_work_dir -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: rename $job_state
by Liu Yinsi 27 Apr '21

27 Apr '21

job_state => report_content See [PATCH lkp-tests 1/3] tests/self-test: send mail when failed to submit job Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index 24ef7bb64..b5e56df79 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -89,7 +89,7 @@ Dear $my_name: Your self-test groud_id=selftest job result is: - $job_state + $report_content Regards Compass-CI -- 2.23.0

1 0

[PATCH compass-ci] container/assistant: add send_account_mail function
by Cao Xueliang 27 Apr '21

27 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 12 +++++++++++ .../assistant/views/send_account_mail.rb | 20 +++++++++++++++++++ lib/assign_account_client.rb | 13 +++++++----- lib/assistant_client.rb | 8 ++++++++ 4 files changed, 48 insertions(+), 5 deletions(-) create mode 100644 container/assistant/views/send_account_mail.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb index 232ec8c..9a96fc0 100755 --- a/container/assistant/routes.rb +++ b/container/assistant/routes.rb @@ -9,6 +9,7 @@ require 'open3' require_relative './views/locate_files' require_relative './views/get_mail_list' +require_relative './views/send_account_mail' configure do set :bind, '0.0.0.0' @@ -37,3 +38,14 @@ get '/get_mail_list/:type' do return [200, result.to_json] end + +post '/send_account_mail' do + begin + data = JSON.parse(request.body.read) + result = send_account_mail(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assistant/views/send_account_mail.rb b/container/assistant/views/send_account_mail.rb new file mode 100644 index 0000000..b96d082 --- /dev/null +++ b/container/assistant/views/send_account_mail.rb @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# # frozen_string_literal: true +# +require_relative "#{ENV['CCI_SRC']}/lib/es_query" +require_relative "#{ENV['CCI_SRC']}/lib/assign_account_client" + +def send_account_mail(user_info) + account_info = ESQuery.new(index: 'accounts').query_by_id(user_info['my_email']) + return {'status'=> true} if account_info + + begin + apply_account = AutoAssignAccount.new(user_info) + apply_account.send_account + rescue + return {'status' => false} + end + + return {'status' => true} +end diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index f551a9d..5aad955 100644 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -4,13 +4,13 @@ # frozen_string_literal: true require 'json' -require 'mail' require 'set' require 'optparse' require 'rest-client' require_relative '../container/defconfig' require_relative 'es_client' require_relative '../container/mail-robot/lib/assign-account-email' +require_relative "#{ENV['CCI_SRC']}/lib/parse_mail_list" names = Set.new %w[ JUMPER_HOST @@ -32,6 +32,7 @@ class AutoAssignAccount @my_info = user_info @my_info_es = {} + @account_info = {} end def update_from_es @@ -59,10 +60,10 @@ class AutoAssignAccount end def update_my_info_from_account_info - account_info = apply_account + @account_info = apply_account - @my_info['my_login_name'] = account_info['my_login_name'] - @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + @my_info['my_login_name'] = @account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << @account_info['my_jumper_pubkey'] unless @account_info['my_jumper_pubkey'].nil? end def store_account_info @@ -72,7 +73,9 @@ class AutoAssignAccount def send_mail @my_info['bisect'] = true - message = build_apply_account_email(@my_info) + default_email = parse_mail_list('delimiter') + @my_info['my_email'] = default_email['to'] unless default_email.empty? + message = build_apply_account_email(@my_info, @account_info, false) %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") end diff --git a/lib/assistant_client.rb b/lib/assistant_client.rb index c951f2b..6774e9e 100644 --- a/lib/assistant_client.rb +++ b/lib/assistant_client.rb @@ -21,4 +21,12 @@ class AssistantClient return JSON.parse(response.body) end + + def send_account_mail(user_info) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_account_mail") + response = resource.post(user_info.to_json) + return nil unless response.code == 200 + + return JSON.parse(response.body) + end end -- 2.23.0

1 0

[PATCH lkp-tests 3/3] jobs: remove multi-docker/qemu
by Liu Yinsi 27 Apr '21

27 Apr '21

multi-qemu and multi-docker will deploy a simply ci env by calling compass-ci/sparrow/install-client, but deploy-cci will deploy a ci env by calling compass-ci/sparrow/install-tiny, they are 2 defferent envs, so remove multi-docker/qemu. testboxes for docker and qemu will be ready by sparrow/8-testbox, 8-testbox already be called by sparrow/install-tiny. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- jobs/self-test.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/jobs/self-test.yaml b/jobs/self-test.yaml index ad2dcd2a2..b0c1c08c3 100644 --- a/jobs/self-test.yaml +++ b/jobs/self-test.yaml @@ -2,5 +2,4 @@ suite: self-test category: functional deploy-cci: -multi-qemu: -multi-docker: +self-test: -- 2.23.0

1 0

[PATCH lkp-tests 2/3] tests/self-test: fix command not found
by Liu Yinsi 27 Apr '21

27 Apr '21

[error] /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 24: es-jobs: command not found Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/self-test | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/self-test b/tests/self-test index 6e4ca7ef2..fb3e87b17 100755 --- a/tests/self-test +++ b/tests/self-test @@ -3,6 +3,8 @@ . $LKP_SRC/lib/email.sh . $LKP_SRC/lib/log.sh +source /etc/profile.d/compass.sh + submit_args=( "testbox=dc-8g cci-depends.benchmark=sshd cci-depends.yaml" "testbox=dc-8g borrow-1h.yaml runtime=1m" -- 2.23.0

1 0

[PATCH lkp-tests 1/3] tests/self-test: send mail when failed to submit job
by Liu Yinsi 27 Apr '21

27 Apr '21

When failed to deploy ci env, some services not ready like es, sched, can't query anything from ES, and email content $job_state is null, so send email when failed to submit job. function submit_one() is redundant, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/self-test | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/tests/self-test b/tests/self-test index cb06b7226..6e4ca7ef2 100755 --- a/tests/self-test +++ b/tests/self-test @@ -1,6 +1,7 @@ #!/bin/bash . $LKP_SRC/lib/email.sh +. $LKP_SRC/lib/log.sh submit_args=( "testbox=dc-8g cci-depends.benchmark=sshd cci-depends.yaml" @@ -8,21 +9,22 @@ submit_args=( "testbox=vm-2p8g borrow-1h.yaml runtime=1m" ) -submit_one() -{ - args="$1" - submit_result=$(submit -m $args group_id=selftest) -} - -submit_all() +submit_job() { for args in "${submit_args[@]}" do - submit_one "$args" + report_content=$(submit -m $args group_id=selftest 2>&1) + echo $report_content | grep -q "\"job_state\":\"submit\"" || + { + log_error "submit $args failed: $report_content" + send_email selftest + exit 1 + } done - job_state=$(es-jobs group_id=selftest | grep job_state) + report_content=$(es-jobs group_id=selftest | grep job_state) + log_info "selftest job result: $report_content" } -submit_all +submit_job send_email selftest -- 2.23.0

1 0

[PATCH lkp-tests] iso2rootfs: some optimize
by Yu Chuan 27 Apr '21

27 Apr '21

1. change dailybuild result rootfs dir name why: - make the dailybuild rootfs directory more meaningful. how: - before: /srv/os/${iso_os}/${iso_arch}/${iso_version}-${TIMESTAMP} - after : /srv/os/${iso_os}/${iso_arch}/${iso_version}-dailybuild-${TIMESTAMP} 2. change sha256sum file why: - unify the file name, then easier to statistics. how: - file name: - before: ${ROOTFS_DES_DIR}/${iso_version}-latest.sha256sum - after : ${ROOTFS_DES_DIR}/iso.sha256sum - file content: - before: 168d2a8c24d20acdf06e67ffee90174b321b52e36afbddfc41602454feb2c09c openEuler-20.03-LTS-aarch64-dvd.iso - after : 168d2a8c24d20acdf06e67ffee90174b321b52e36afbddfc41602454feb2c09c Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 06c8565110ab..e8694a73e807 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -318,6 +318,8 @@ download_compass_ci() config_rootfs_dir() { ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") + [ -n "${dailybuild_iso_url_file}" ] && ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-dailybuild-$(date "+%Y-%m-%d-%H-%M-%S") + [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -592,7 +594,7 @@ update_dailybuild_soft_link() [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}" + echo $SHA256SUM_NET > "${ROOTFS_DES_DIR}/iso.sha256sum" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && -- 2.23.0

2 2

[PATCH lkp-tests 2/3] install-iso-pre: change the logic of ks generate and upload
by Yu Chuan 27 Apr '21

27 Apr '21

[Why] 1. 'key difference' of ks for different physical machine: - http install source. - install target disk/partition. - working nic. 2. 'other ks content' for different os is different. so the change is as following: before: 'other ks content' is all same. after : add a httpd dir, which stores the 'other ks content' for different os, then they only need generate the 'key difference' related to current physical machine, 'other ks content' can download from httpd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 43 +++++++++---------------------------------- 1 file changed, 9 insertions(+), 34 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c89895a994cc..f9ff9ebe9e94 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -91,6 +91,13 @@ prepare_lv() create_lv } +append_ks_template() +{ + wget -O ${ks_file}.tmp http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/install/ks/template/${iso_os}-${iso_arch}-${iso_version}.ks + cat ${ks_file}.tmp >> $ks_file + rm -rf ${ks_file}.tmp +} + generate_ks() { log_info "start generate $ks_file" @@ -112,41 +119,9 @@ generate_ks() bootloader --location=none logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting - # Root password - rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} - - # Reboot after installation - reboot --eject - - # Use text mode install - text - - # Keyboard layouts - keyboard --vckeymap=us --xlayouts='us' - - # System language - lang en_US.UTF-8 - - # Do not configure the X Window System - skipx - - # System services - services --enabled="chronyd" - - # System timezone - timezone Asia/Shanghai --isUtc - - %packages - @^minimal-environment - - %end - - %anaconda - pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty - pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok - pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty - %end EOF + + append_ks_template } upload_ks() -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/web_backend.rb: show OS by order on web
by Li Yuanchao 27 Apr '21

27 Apr '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 576b922..da3c3c5 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -97,10 +97,10 @@ def compare_candidates_body query_conditions: { suite: all_suite, OS: [ - { os: 'openeuler', os_version: ['1.0', '20.03'] }, + { os: 'archlinux', os_version: ['5.5.0-1'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1'] }, { os: 'debian', os_version: %w[10 sid] }, - { os: 'archlinux', os_version: ['5.5.0-1'] } + { os: 'openeuler', os_version: ['1.0', '20.03'] } ], os_arch: %w[aarch64 x86_64], tbox_group: all_tbox_group -- 2.23.0

1 0

[PATCH lkp-tests 3/3] install-iso: fix some bugs
by Yu Chuan 27 Apr '21

27 Apr '21

[What Bugs] 1. optimize name: reboot.cgz => reboot-after-sshd.cgz 2. nfs mount result dir: - before: ${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} - after : ${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} 3. exit directory before umount Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 6 +++--- tests/install-iso-sync-rootfs | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index f9ff9ebe9e94..16639e437d71 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -203,8 +203,8 @@ submit_install_iso_firstboot_yaml() custom_ipxe="#!ipxe\\\\n" custom_ipxe="${custom_ipxe}\\\\ndhcp" custom_ipxe="${custom_ipxe}\\\\ninitrd http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/initrd.lkp" - custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot.cgz" - custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot.cgz local rw" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot-after-sshd.cgz" + custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot-after-sshd.cgz local rw" custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" $firstboot_yaml @@ -232,7 +232,7 @@ submit_install_iso_sync_rootfs_yaml() install-iso-sync-rootfs.NFS_ROOT_DIR=${NFS_ROOT_DIR} \ install-iso-sync-rootfs.ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ install-iso-sync-rootfs.VG_NAME=${VG_NAME} \ - testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" + testbox=$testbox || die "submit job failed: ${sync_rootfs_yaml}" } main() diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index 34096229ccfb..b74c7bed2044 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -59,7 +59,7 @@ sync_rootfs_to_nfs_server() local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} mkdir -p ${nfs_rootfs_mount_point} - mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" # activate the lvm vgchange -ay @@ -78,7 +78,7 @@ sync_rootfs_to_nfs_server() cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" # umount nfs rootfs - umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + cd / && umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" # umount local logical volume umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" -- 2.23.0

2 1

[PATCH v5 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. examples: a/AvxToNeon/AvxToNeon aur-a/a2png/a2png m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index cbf6e5c..0c1f4fa 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-zA-Z0-9][\w\-]*[a-zA-Z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: show OS by order on web
by Li Yuanchao 27 Apr '21

27 Apr '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index dde64c0..03db8f6 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -97,10 +97,10 @@ def compare_candidates_body query_conditions: { suite: all_suite, OS: [ - { os: 'openeuler', os_version: ['1.0', '20.03'] }, + { os: 'archlinux', os_version: ['5.5.0-1'] } { os: 'centos', os_version: ['7.6', '7.8', '8.1'] }, { os: 'debian', os_version: %w[10 sid] }, - { os: 'archlinux', os_version: ['5.5.0-1'] } + { os: 'openeuler', os_version: ['1.0', '20.03'] }, ], os_arch: %w[aarch64 x86_64], tbox_group: all_tbox_group -- 2.23.0

1 0

[PATCH v4 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 91287b4..dde64c0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-z0-9][a-z0-9\-_]*[a-z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 91287b4..0bfc6c9 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ /^([a-z0-9]([a-z0-9\-\_]*[a-z0-9])*(\/[a-z0-9][a-z0-9\-\_]*[a-z0-9]){2})$/ + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow/3-code: delete special symbols in vars values
by Liu Yinsi 27 Apr '21

27 Apr '21

[why] special symbols will be parsed in shell command "docker build -t --build-arg $PASSWORD ...", caused failed to build images, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index b68d9e67..15d9bc45 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -190,7 +190,7 @@ auto-submit: EOF [[ -f /etc/compass-ci/passwd.yaml ]] || { -passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9-=+@#_:/[](){}$%&.^*\!" | head -c 10;echo) +passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9" | head -c 10; echo) cat > /etc/compass-ci/passwd.yaml <<EOF ES_USER: root -- 2.23.0

2 2

[PATCH compass-ci 1/2] sparrow/1-storage: set permission /etc/compass-ci/password.yaml 660
by Liu Yinsi 27 Apr '21

27 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/1-storage/tiny | 1 + 2 files changed, 2 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 3c229c5a..3a8bfdaa 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -44,5 +44,6 @@ chgrp team /srv/cci/serial/logs chgrp team /srv/dc chmod g+ws /srv/dc +chmod 660 /etc/compass-ci/password.yaml find /tftpboot -type d | xargs chmod 775 diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index c7636b02..1dbe8a32 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -22,6 +22,7 @@ dirs=( /c /c/git-repos /etc/qemu + /etc/compass-ci/password.yaml /srv/cci/Maildir/.compass-ci/new /srv/cci/Maildir/.compass-ci/cur /srv/cci/Maildir/.compass-ci/sent -- 2.23.0

2 2

2024

2023

2022

2021

2020

Compass-ci