- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/rpmbuild-pkg: change the way obtaining spec file
by Li Ping 28 Apr '21

28 Apr '21

[why] tests/rpmbuild-pkg: package_name=${upstream_repo##*/} when the package name is inconsistent with the repo name, it will cause the error like: + grep http -rl /root/rpmbuild/SPECS/zstd-test.spec grep: /root/rpmbuild/SPECS/zstd-test.spec: No such file or directory + spectool -g -R /root/rpmbuild/SPECS/zstd-test.spec Can't open '/root/rpmbuild/SPECS/zstd-test.spec': No such file or directory at /usr/bin/spectool line 314. Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 5c694b9d4..3dd75b896 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -49,7 +49,7 @@ init_workspace() download_upstream_repo() { - local url="git://${GIT_SERVER}/openeuler/${upstream_repo}" + local url="git://${GIT_SERVER}/${upstream_dir}/${upstream_repo}" git clone "$url" || die "clone git repo ${package_name} failed: $url" cd "$package_name" || exit @@ -75,14 +75,14 @@ build_rpm() [ -n "$package_name" ] && { # HTTP is proxy cache friendly - sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/${package_name}.spec")" + sed -i 's/^$Source[^ ]*:[ \t]*$https/\1http/g' "$(grep http -rl "$spec_dir/"*.spec)" } # Install build depends yum-builddep -y "$spec_dir"/*.spec || die "failed to solve dependencies" # Download tar.gz to default path ${HOME}/rpmbuild/SOURCE [ -n "$package_name" ] && { - spectool -g -R "$spec_dir/${package_name}.spec" || die "failed to download source file" + spectool -g -R "$spec_dir/"*.spec || die "failed to download source file" } # Building rpm or srpm packages rpmbuild -ba "$spec_dir"/*.spec || die "failed to build rpms" -- 2.23.0

1 0

[PATCH v6 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 28 Apr '21

28 Apr '21

To make sure git_repos displayed on the web are right. examples of git_repo: a/AvxToNeon/AvxToNeon 0/caterqiu/001 aur-a/a2png/a2png m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 9e899d9..e305ac6 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -400,6 +400,14 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + + # What does this regular expression want: + # 1. a three-segment structure "xxx/xxx/xxx" + # 2. there can be '-' or '_' in every segment, but can't be at first or last of the segment. + # 3. the first segment can have lowercase of letters or numbers in it. + # 4. the other two segments can have letters(lowercase or uppercase) or numbers in it. + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-zA-Z0-9][\w\-]*[a-zA-Z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH lkp-tests] jobs/send-email.yaml: change the default value.
by Li Ping 28 Apr '21

28 Apr '21

the default value of email subject should not be job_failed, and there is already a send-email-on-fail.yaml. Signed-off-by: Li Ping <1477412247(a)qq.com> --- jobs/send-email.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/jobs/send-email.yaml b/jobs/send-email.yaml index 8f8b48b96..2689634c2 100644 --- a/jobs/send-email.yaml +++ b/jobs/send-email.yaml @@ -3,4 +3,4 @@ # values of $subject are limited to job_failed/job_debug/borrow_success/borrow_failed. send_email: - subject: job_failed + subject: -- 2.23.0

1 0

[PATCH compass-ci 2/2] kibana/dashboard_import: import kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can import kibana dashboards by json files. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_import | 45 +++++++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100755 container/kibana/dashboard_import diff --git a/container/kibana/dashboard_import b/container/kibana/dashboard_import new file mode 100755 index 0000000..670fa3f --- /dev/null +++ b/container/kibana/dashboard_import @@ -0,0 +1,45 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require 'pathname' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +files = [] +ARGV.each do |f| + files >> f +end + +if files.empty? + profile = Pathname.new(File.dirname(__FILE__)).realpath + files << profile + 'data.json' + files << profile + 'resource.json' + files << profile + 'all.json' +end +puts files + +cmd = "curl -XPOST -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/import" +cmd += " -H 'Content-Type: application/json' -H 'kbn-xsrf: reporting'" + +files.each do |f| + new_cmd = cmd + new_cmd += " -d @#{f}" + puts new_cmd + system new_cmd +end -- 2.23.0

2 1

[PATCH compass-ci] src/lib/web_backend.rb: modify parameter list for web compare
by Li Yuanchao 28 Apr '21

28 Apr '21

The parameter list for web compare need to change, as tbox_groups have $user at the back. Suite and tbox_group whose count < 2 need not to show, cause have no one to compare. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 25 +++++++++++-------------- 1 file changed, 11 insertions(+), 14 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index da3c3c5..a9ab091 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -45,16 +45,11 @@ def es_count(query) ES_CLIENT.count(index: 'jobs*', body: query)['count'] end -# "vm-2p8g--212" remove "--212" -# "vm-2p8g--zzz" remove "--zzz" -# "vm-git-bisect" don't remove "-bisect" +# delete $user after '--' or '.' or '~' def filter_tbox_group(es_result) result = Set.new es_result.each do |r| - if r =~ /(^.+--.+$)|(^vm-.*-\d\w*-(([a-zA-Z]+)|(\d+))$)/ - index = r.index('--') || r.rindex('-') - r = r[0, index] - end + r = r.gsub(/(--|\.|~).*$/, '') result.add r end result.to_a @@ -70,10 +65,10 @@ def all_suite size: 0 } es_result = es_query(query)['aggregations']['all_suite']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } - - es_result + result_second = [] + es_result.each { |e| result_second << e if e['doc_count'] > 1 } + result_second.sort_by! { |h| h['doc_count'] } + result_second.reverse!.map! { |x| x['key'] } end def all_tbox_group @@ -86,10 +81,12 @@ def all_tbox_group size: 0 } es_result = es_query(query)['aggregations']['all_tbox_group']['buckets'] - es_result.sort_by! { |h| h['doc_count'] } - es_result.reverse!.map! { |x| x['key'] } + result_second = [] + es_result.each { |e| result_second << e if e['doc_count'] > 1 } + result_second.sort_by! { |h| h['doc_count'] } + result_second.reverse!.map! { |x| x['key'] } - filter_tbox_group(es_result) + filter_tbox_group(result_second) end def compare_candidates_body -- 2.23.0

2 1

[PATCH v2 compass-ci] sbin/es-jobs-mapping.sh: add field
by Lu Weitao 28 Apr '21

28 Apr '21

add nr_run(integer), author_email(keyword) Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-jobs-mapping.sh | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 67b1ed14..0c2fc62d 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -148,6 +148,12 @@ else "upstream_commit": { "type": "keyword" }, + "base_commit": { + "type": "keyword" + }, + "nr_run": { + "type": "integer" + }, "my_email": { "type": "keyword" }, -- 2.23.0

2 2

[PATCH compass-ci 1/2] kibana/dashboard_export: exporting kibana dashboards
by Wu Zhende 28 Apr '21

28 Apr '21

Use this script can export kibana dashboards by ids. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/kibana/dashboard_export | 37 +++++++++++++++++++++++++++++++ 1 file changed, 37 insertions(+) create mode 100755 container/kibana/dashboard_export diff --git a/container/kibana/dashboard_export b/container/kibana/dashboard_export new file mode 100755 index 0000000..d6e0054 --- /dev/null +++ b/container/kibana/dashboard_export @@ -0,0 +1,37 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'set' +require_relative '../defconfig.rb' + +names = Set.new %w[ + KIBANA_HOST + KIBANA_PORT + ES_USER + ES_PASSWORD +] + +defaults = relevant_defaults(names) +KIBANA_HOST = defaults['KIBANA_HOST'] || '172.17.0.1' +KIBANA_PORT = defaults['KIBANA_PORT'] || '20017' + +service_authentication = relevant_service_authentication(names) + +dashboard_ids = [] +ARGV.each do |i| + dashboard_ids << i +end + +dashboard_ids = %w[data resource all] if dashboard_ids.empty? + +cmd = "curl -u #{service_authentication['ES_USER']}:#{service_authentication['ES_PASSWORD']}" +cmd += " http://#{KIBANA_HOST}:#{KIBANA_PORT}/api/kibana/dashboards/export?dashboard=" + +dashboard_ids.each do |id| + new_cmd = cmd + new_cmd += "#{id} >> #{id}.json" + puts new_cmd + system new_cmd +end -- 2.23.0

1 0

[PATCH v2 lkp-tests 3/3] iso2rootfs: some optimize
by Yu Chuan 27 Apr '21

27 Apr '21

1. change dailybuild result rootfs dir name why: - make the dailybuild rootfs directory more meaningful. how: - before: /srv/os/${iso_os}/${iso_arch}/${iso_version}-${TIMESTAMP} - after : /srv/os/${iso_os}/${iso_arch}/${iso_version}-dailybuild-${TIMESTAMP} 2. change sha256sum file why: - unify the file name, then easier to statistics. how: - file name: - before: ${ROOTFS_DES_DIR}/${iso_version}-latest.sha256sum - after : ${ROOTFS_DES_DIR}/SHA256SUM Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 06c8565110ab..ca972b2399f6 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -318,6 +318,8 @@ download_compass_ci() config_rootfs_dir() { ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") + [ -n "${dailybuild_iso_url_file}" ] && ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-dailybuild-$(date "+%Y-%m-%d-%H-%M-%S") + [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -592,7 +594,7 @@ update_dailybuild_soft_link() [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}" + cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}/SHA256SUM" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && -- 2.23.0

1 0

[PATCH v2 lkp-tests 2/3] install-iso: fix some bugs
by Yu Chuan 27 Apr '21

27 Apr '21

[What Bugs] 1. optimize name: reboot.cgz => reboot-after-systemd-logind.cgz 2. nfs mount result dir: - before: ${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} - after : ${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} 3. exit directory before umount Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 6 +++--- tests/install-iso-sync-rootfs | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c51b3784b5aa..481b99e35df4 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -196,8 +196,8 @@ submit_install_iso_firstboot_yaml() custom_ipxe="#!ipxe\\\\n" custom_ipxe="${custom_ipxe}\\\\ndhcp" custom_ipxe="${custom_ipxe}\\\\ninitrd http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/initrd.lkp" - custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot.cgz" - custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot.cgz local rw" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot-after-systemd-logind.cgz" + custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot-after-systemd-logind.cgz local rw" custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" $firstboot_yaml @@ -225,7 +225,7 @@ submit_install_iso_sync_rootfs_yaml() install-iso-sync-rootfs.NFS_ROOT_DIR=${NFS_ROOT_DIR} \ install-iso-sync-rootfs.ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ install-iso-sync-rootfs.VG_NAME=${VG_NAME} \ - testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" + testbox=$testbox || die "submit job failed: ${sync_rootfs_yaml}" } main() diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index 34096229ccfb..b74c7bed2044 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -59,7 +59,7 @@ sync_rootfs_to_nfs_server() local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} mkdir -p ${nfs_rootfs_mount_point} - mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" # activate the lvm vgchange -ay @@ -78,7 +78,7 @@ sync_rootfs_to_nfs_server() cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" # umount nfs rootfs - umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + cd / && umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" # umount local logical volume umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" -- 2.23.0

1 0

[PATCH v2 lkp-tests 1/3] install-iso-pre: change the logic of ks generate and upload
by Yu Chuan 27 Apr '21

27 Apr '21

[Why] 1. 'key difference' of ks for different physical machine: - Choice Install Source - Choice Install Partition - Choice working Nic 2. 'other ks content' for different os is different. so the change is as following: before: 'other ks content' is all same. after : add a httpd dir, which stores the 'other ks content' for different os, then they only need generate the 'key difference' related to current physical machine, 'other ks content' can download from httpd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 36 ++---------------------------------- 1 file changed, 2 insertions(+), 34 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c89895a994cc..c51b3784b5aa 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -112,41 +112,9 @@ generate_ks() bootloader --location=none logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting - # Root password - rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} - - # Reboot after installation - reboot --eject - - # Use text mode install - text - - # Keyboard layouts - keyboard --vckeymap=us --xlayouts='us' - - # System language - lang en_US.UTF-8 - - # Do not configure the X Window System - skipx - - # System services - services --enabled="chronyd" - - # System timezone - timezone Asia/Shanghai --isUtc - - %packages - @^minimal-environment - - %end - - %anaconda - pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty - pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok - pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty - %end EOF + + curl -s http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/install/ks/template/${iso_os}-${iso_arch}-${iso_version}.ks >> $ks_file } upload_ks() -- 2.23.0

1 0

[PATCH compass-ci] lib/mail_bisect_result: use assistant client send account mail
by Cao Xueliang 27 Apr '21

27 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/mail_bisect_result.rb | 7 +------ 1 file changed, 1 insertion(+), 6 deletions(-) diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 38828a3..1112f3b 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -84,12 +84,7 @@ class MailBisectResult 'my_name' => @git_commit.author_name, 'my_commit_url' => "#{@git_commit.url}/commit/#{@commit_id}" } - - account_info = ESQuery.new(index: 'accounts').query_by_id(@to) - return if account_info - - apply_account = AutoAssignAccount.new(user_info) - apply_account.send_account + AssistantClient.new.send_account_mail(user_info) end def rm_work_dir -- 2.23.0

1 0

[PATCH lkp-tests] lib/email.sh: rename $job_state
by Liu Yinsi 27 Apr '21

27 Apr '21

job_state => report_content See [PATCH lkp-tests 1/3] tests/self-test: send mail when failed to submit job Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- lib/email.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/email.sh b/lib/email.sh index 24ef7bb64..b5e56df79 100755 --- a/lib/email.sh +++ b/lib/email.sh @@ -89,7 +89,7 @@ Dear $my_name: Your self-test groud_id=selftest job result is: - $job_state + $report_content Regards Compass-CI -- 2.23.0

1 0

[PATCH compass-ci] container/assistant: add send_account_mail function
by Cao Xueliang 27 Apr '21

27 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assistant/routes.rb | 12 +++++++++++ .../assistant/views/send_account_mail.rb | 20 +++++++++++++++++++ lib/assign_account_client.rb | 13 +++++++----- lib/assistant_client.rb | 8 ++++++++ 4 files changed, 48 insertions(+), 5 deletions(-) create mode 100644 container/assistant/views/send_account_mail.rb diff --git a/container/assistant/routes.rb b/container/assistant/routes.rb index 232ec8c..9a96fc0 100755 --- a/container/assistant/routes.rb +++ b/container/assistant/routes.rb @@ -9,6 +9,7 @@ require 'open3' require_relative './views/locate_files' require_relative './views/get_mail_list' +require_relative './views/send_account_mail' configure do set :bind, '0.0.0.0' @@ -37,3 +38,14 @@ get '/get_mail_list/:type' do return [200, result.to_json] end + +post '/send_account_mail' do + begin + data = JSON.parse(request.body.read) + result = send_account_mail(data) + rescue StandardError => e + return [400, e.backtrace.inspect] + end + + return [200, result.to_json] +end diff --git a/container/assistant/views/send_account_mail.rb b/container/assistant/views/send_account_mail.rb new file mode 100644 index 0000000..b96d082 --- /dev/null +++ b/container/assistant/views/send_account_mail.rb @@ -0,0 +1,20 @@ +# SPDX-License-Identifier: MulanPSL-2.0+ +# # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# # frozen_string_literal: true +# +require_relative "#{ENV['CCI_SRC']}/lib/es_query" +require_relative "#{ENV['CCI_SRC']}/lib/assign_account_client" + +def send_account_mail(user_info) + account_info = ESQuery.new(index: 'accounts').query_by_id(user_info['my_email']) + return {'status'=> true} if account_info + + begin + apply_account = AutoAssignAccount.new(user_info) + apply_account.send_account + rescue + return {'status' => false} + end + + return {'status' => true} +end diff --git a/lib/assign_account_client.rb b/lib/assign_account_client.rb index f551a9d..5aad955 100644 --- a/lib/assign_account_client.rb +++ b/lib/assign_account_client.rb @@ -4,13 +4,13 @@ # frozen_string_literal: true require 'json' -require 'mail' require 'set' require 'optparse' require 'rest-client' require_relative '../container/defconfig' require_relative 'es_client' require_relative '../container/mail-robot/lib/assign-account-email' +require_relative "#{ENV['CCI_SRC']}/lib/parse_mail_list" names = Set.new %w[ JUMPER_HOST @@ -32,6 +32,7 @@ class AutoAssignAccount @my_info = user_info @my_info_es = {} + @account_info = {} end def update_from_es @@ -59,10 +60,10 @@ class AutoAssignAccount end def update_my_info_from_account_info - account_info = apply_account + @account_info = apply_account - @my_info['my_login_name'] = account_info['my_login_name'] - @my_info['my_ssh_pubkey'] << account_info['my_jumper_pubkey'] unless account_info['my_jumper_pubkey'].nil? + @my_info['my_login_name'] = @account_info['my_login_name'] + @my_info['my_ssh_pubkey'] << @account_info['my_jumper_pubkey'] unless @account_info['my_jumper_pubkey'].nil? end def store_account_info @@ -72,7 +73,9 @@ class AutoAssignAccount def send_mail @my_info['bisect'] = true - message = build_apply_account_email(@my_info) + default_email = parse_mail_list('delimiter') + @my_info['my_email'] = default_email['to'] unless default_email.empty? + message = build_apply_account_email(@my_info, @account_info, false) %x(curl -XPOST "#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text" -d "#{message}") end diff --git a/lib/assistant_client.rb b/lib/assistant_client.rb index c951f2b..6774e9e 100644 --- a/lib/assistant_client.rb +++ b/lib/assistant_client.rb @@ -21,4 +21,12 @@ class AssistantClient return JSON.parse(response.body) end + + def send_account_mail(user_info) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/send_account_mail") + response = resource.post(user_info.to_json) + return nil unless response.code == 200 + + return JSON.parse(response.body) + end end -- 2.23.0

1 0

[PATCH lkp-tests 3/3] jobs: remove multi-docker/qemu
by Liu Yinsi 27 Apr '21

27 Apr '21

multi-qemu and multi-docker will deploy a simply ci env by calling compass-ci/sparrow/install-client, but deploy-cci will deploy a ci env by calling compass-ci/sparrow/install-tiny, they are 2 defferent envs, so remove multi-docker/qemu. testboxes for docker and qemu will be ready by sparrow/8-testbox, 8-testbox already be called by sparrow/install-tiny. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- jobs/self-test.yaml | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/jobs/self-test.yaml b/jobs/self-test.yaml index ad2dcd2a2..b0c1c08c3 100644 --- a/jobs/self-test.yaml +++ b/jobs/self-test.yaml @@ -2,5 +2,4 @@ suite: self-test category: functional deploy-cci: -multi-qemu: -multi-docker: +self-test: -- 2.23.0

1 0

[PATCH lkp-tests 2/3] tests/self-test: fix command not found
by Liu Yinsi 27 Apr '21

27 Apr '21

[error] /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 14: submit: command not found /lkp/lkp/src/tests/self-test: line 24: es-jobs: command not found Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/self-test | 2 ++ 1 file changed, 2 insertions(+) diff --git a/tests/self-test b/tests/self-test index 6e4ca7ef2..fb3e87b17 100755 --- a/tests/self-test +++ b/tests/self-test @@ -3,6 +3,8 @@ . $LKP_SRC/lib/email.sh . $LKP_SRC/lib/log.sh +source /etc/profile.d/compass.sh + submit_args=( "testbox=dc-8g cci-depends.benchmark=sshd cci-depends.yaml" "testbox=dc-8g borrow-1h.yaml runtime=1m" -- 2.23.0

1 0

[PATCH lkp-tests 1/3] tests/self-test: send mail when failed to submit job
by Liu Yinsi 27 Apr '21

27 Apr '21

When failed to deploy ci env, some services not ready like es, sched, can't query anything from ES, and email content $job_state is null, so send email when failed to submit job. function submit_one() is redundant, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- tests/self-test | 22 ++++++++++++---------- 1 file changed, 12 insertions(+), 10 deletions(-) diff --git a/tests/self-test b/tests/self-test index cb06b7226..6e4ca7ef2 100755 --- a/tests/self-test +++ b/tests/self-test @@ -1,6 +1,7 @@ #!/bin/bash . $LKP_SRC/lib/email.sh +. $LKP_SRC/lib/log.sh submit_args=( "testbox=dc-8g cci-depends.benchmark=sshd cci-depends.yaml" @@ -8,21 +9,22 @@ submit_args=( "testbox=vm-2p8g borrow-1h.yaml runtime=1m" ) -submit_one() -{ - args="$1" - submit_result=$(submit -m $args group_id=selftest) -} - -submit_all() +submit_job() { for args in "${submit_args[@]}" do - submit_one "$args" + report_content=$(submit -m $args group_id=selftest 2>&1) + echo $report_content | grep -q "\"job_state\":\"submit\"" || + { + log_error "submit $args failed: $report_content" + send_email selftest + exit 1 + } done - job_state=$(es-jobs group_id=selftest | grep job_state) + report_content=$(es-jobs group_id=selftest | grep job_state) + log_info "selftest job result: $report_content" } -submit_all +submit_job send_email selftest -- 2.23.0

1 0

[PATCH lkp-tests] iso2rootfs: some optimize
by Yu Chuan 27 Apr '21

27 Apr '21

1. change dailybuild result rootfs dir name why: - make the dailybuild rootfs directory more meaningful. how: - before: /srv/os/${iso_os}/${iso_arch}/${iso_version}-${TIMESTAMP} - after : /srv/os/${iso_os}/${iso_arch}/${iso_version}-dailybuild-${TIMESTAMP} 2. change sha256sum file why: - unify the file name, then easier to statistics. how: - file name: - before: ${ROOTFS_DES_DIR}/${iso_version}-latest.sha256sum - after : ${ROOTFS_DES_DIR}/iso.sha256sum - file content: - before: 168d2a8c24d20acdf06e67ffee90174b321b52e36afbddfc41602454feb2c09c openEuler-20.03-LTS-aarch64-dvd.iso - after : 168d2a8c24d20acdf06e67ffee90174b321b52e36afbddfc41602454feb2c09c Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 06c8565110ab..e8694a73e807 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -318,6 +318,8 @@ download_compass_ci() config_rootfs_dir() { ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-$(date "+%Y-%m-%d-%H-%M-%S") + [ -n "${dailybuild_iso_url_file}" ] && ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${iso_version}-dailybuild-$(date "+%Y-%m-%d-%H-%M-%S") + [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -592,7 +594,7 @@ update_dailybuild_soft_link() [ -n "${dailybuild_iso_url_file}" ] || return 0 mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" - cp "${CHECKSUM_FILE_CACHE}" "${ROOTFS_DES_DIR}" + echo $SHA256SUM_NET > "${ROOTFS_DES_DIR}/iso.sha256sum" local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && -- 2.23.0

2 2

[PATCH lkp-tests 2/3] install-iso-pre: change the logic of ks generate and upload
by Yu Chuan 27 Apr '21

27 Apr '21

[Why] 1. 'key difference' of ks for different physical machine: - http install source. - install target disk/partition. - working nic. 2. 'other ks content' for different os is different. so the change is as following: before: 'other ks content' is all same. after : add a httpd dir, which stores the 'other ks content' for different os, then they only need generate the 'key difference' related to current physical machine, 'other ks content' can download from httpd. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 43 +++++++++---------------------------------- 1 file changed, 9 insertions(+), 34 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index c89895a994cc..f9ff9ebe9e94 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -91,6 +91,13 @@ prepare_lv() create_lv } +append_ks_template() +{ + wget -O ${ks_file}.tmp http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/install/ks/template/${iso_os}-${iso_arch}-${iso_version}.ks + cat ${ks_file}.tmp >> $ks_file + rm -rf ${ks_file}.tmp +} + generate_ks() { log_info "start generate $ks_file" @@ -112,41 +119,9 @@ generate_ks() bootloader --location=none logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting - # Root password - rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} - - # Reboot after installation - reboot --eject - - # Use text mode install - text - - # Keyboard layouts - keyboard --vckeymap=us --xlayouts='us' - - # System language - lang en_US.UTF-8 - - # Do not configure the X Window System - skipx - - # System services - services --enabled="chronyd" - - # System timezone - timezone Asia/Shanghai --isUtc - - %packages - @^minimal-environment - - %end - - %anaconda - pwpolicy root --minlen=8 --minquality=1 --notstrict --nochanges --notempty - pwpolicy user --minlen=8 --minquality=1 --notstrict --nochanges --emptyok - pwpolicy luks --minlen=8 --minquality=1 --notstrict --nochanges --notempty - %end EOF + + append_ks_template } upload_ks() -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/web_backend.rb: show OS by order on web
by Li Yuanchao 27 Apr '21

27 Apr '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 576b922..da3c3c5 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -97,10 +97,10 @@ def compare_candidates_body query_conditions: { suite: all_suite, OS: [ - { os: 'openeuler', os_version: ['1.0', '20.03'] }, + { os: 'archlinux', os_version: ['5.5.0-1'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1'] }, { os: 'debian', os_version: %w[10 sid] }, - { os: 'archlinux', os_version: ['5.5.0-1'] } + { os: 'openeuler', os_version: ['1.0', '20.03'] } ], os_arch: %w[aarch64 x86_64], tbox_group: all_tbox_group -- 2.23.0

1 0

[PATCH lkp-tests 3/3] install-iso: fix some bugs
by Yu Chuan 27 Apr '21

27 Apr '21

[What Bugs] 1. optimize name: reboot.cgz => reboot-after-sshd.cgz 2. nfs mount result dir: - before: ${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} - after : ${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} 3. exit directory before umount Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 6 +++--- tests/install-iso-sync-rootfs | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index f9ff9ebe9e94..16639e437d71 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -203,8 +203,8 @@ submit_install_iso_firstboot_yaml() custom_ipxe="#!ipxe\\\\n" custom_ipxe="${custom_ipxe}\\\\ndhcp" custom_ipxe="${custom_ipxe}\\\\ninitrd http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/initrd.lkp" - custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot.cgz" - custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot.cgz local rw" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot-after-sshd.cgz" + custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot-after-sshd.cgz local rw" custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" $firstboot_yaml @@ -232,7 +232,7 @@ submit_install_iso_sync_rootfs_yaml() install-iso-sync-rootfs.NFS_ROOT_DIR=${NFS_ROOT_DIR} \ install-iso-sync-rootfs.ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ install-iso-sync-rootfs.VG_NAME=${VG_NAME} \ - testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" + testbox=$testbox || die "submit job failed: ${sync_rootfs_yaml}" } main() diff --git a/tests/install-iso-sync-rootfs b/tests/install-iso-sync-rootfs index 34096229ccfb..b74c7bed2044 100755 --- a/tests/install-iso-sync-rootfs +++ b/tests/install-iso-sync-rootfs @@ -59,7 +59,7 @@ sync_rootfs_to_nfs_server() local nfs_rootfs_mount_point=/tmp/nfs_rootfs_${ROOTFS_SUFFIX} mkdir -p ${nfs_rootfs_mount_point} - mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${os}/${os_arch}/${os_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" + mount -t nfs ${NFS_SERVER_HOST}:${NFS_ROOT_DIR}/${iso_os}/${iso_arch}/${iso_version}-iso-${ROOTFS_SUFFIX} ${nfs_rootfs_mount_point} || die "mount nfs rootfs failed" # activate the lvm vgchange -ay @@ -78,7 +78,7 @@ sync_rootfs_to_nfs_server() cd ${nfs_rootfs_mount_point} && mkdir -p "${SKIP_SYNC_DIRS[@]}" # umount nfs rootfs - umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" + cd / && umount $nfs_rootfs_mount_point || log_warn "umount failed: nfs_rootfs_mount_point" # umount local logical volume umount $lv_rootfs_mount_point || log_warn "umount failed: $lv_rootfs_mount_point" -- 2.23.0

2 1

[PATCH v5 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. examples: a/AvxToNeon/AvxToNeon aur-a/a2png/a2png m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index cbf6e5c..0c1f4fa 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-zA-Z0-9][\w\-]*[a-zA-Z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: show OS by order on web
by Li Yuanchao 27 Apr '21

27 Apr '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index dde64c0..03db8f6 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -97,10 +97,10 @@ def compare_candidates_body query_conditions: { suite: all_suite, OS: [ - { os: 'openeuler', os_version: ['1.0', '20.03'] }, + { os: 'archlinux', os_version: ['5.5.0-1'] } { os: 'centos', os_version: ['7.6', '7.8', '8.1'] }, { os: 'debian', os_version: %w[10 sid] }, - { os: 'archlinux', os_version: ['5.5.0-1'] } + { os: 'openeuler', os_version: ['1.0', '20.03'] }, ], os_arch: %w[aarch64 x86_64], tbox_group: all_tbox_group -- 2.23.0

1 0

[PATCH v4 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 91287b4..dde64c0 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ %r{^([a-z0-9]([a-z0-9\-_]*[a-z0-9])*(/[a-z0-9][a-z0-9\-_]*[a-z0-9]){2})$} + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH v3 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 91287b4..0bfc6c9 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ /^([a-z0-9]([a-z0-9\-\_]*[a-z0-9])*(\/[a-z0-9][a-z0-9\-\_]*[a-z0-9]){2})$/ + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH compass-ci 2/2] sparrow/3-code: delete special symbols in vars values
by Liu Yinsi 27 Apr '21

27 Apr '21

[why] special symbols will be parsed in shell command "docker build -t --build-arg $PASSWORD ...", caused failed to build images, so delete. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index b68d9e67..15d9bc45 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -190,7 +190,7 @@ auto-submit: EOF [[ -f /etc/compass-ci/passwd.yaml ]] || { -passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9-=+@#_:/[](){}$%&.^*\!" | head -c 10;echo) +passwd=$(< /dev/urandom tr -dc "a-zA-Z0-9" | head -c 10; echo) cat > /etc/compass-ci/passwd.yaml <<EOF ES_USER: root -- 2.23.0

2 2

[PATCH compass-ci 1/2] sparrow/1-storage: set permission /etc/compass-ci/password.yaml 660
by Liu Yinsi 27 Apr '21

27 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 1 + sparrow/1-storage/tiny | 1 + 2 files changed, 2 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index 3c229c5a..3a8bfdaa 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -44,5 +44,6 @@ chgrp team /srv/cci/serial/logs chgrp team /srv/dc chmod g+ws /srv/dc +chmod 660 /etc/compass-ci/password.yaml find /tftpboot -type d | xargs chmod 775 diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index c7636b02..1dbe8a32 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -22,6 +22,7 @@ dirs=( /c /c/git-repos /etc/qemu + /etc/compass-ci/password.yaml /srv/cci/Maildir/.compass-ci/new /srv/cci/Maildir/.compass-ci/cur /srv/cci/Maildir/.compass-ci/sent -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 91287b4..066b781 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ /^([a-z0-9])/ + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH compass-ci] src/lib/web_backend.rb: filter git_repos
by Li Yuanchao 27 Apr '21

27 Apr '21

To make sure git_repos displayed on the web are right. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 2 ++ 1 file changed, 2 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 91287b4..452e94a 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -403,6 +403,8 @@ def query_repos(must, from: 0, size: 1) repos = [] result['hits']['hits'].each do |r| r = r['_source'] + next unless r['git_repo'] =~ /^([a-z]\/)/ + repos << { git_url: get_repo_url(r['url']), git_repo: r['git_repo'] -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-*-mapping.sh: configuring ES authentication information
by Wu Zhende 27 Apr '21

27 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-regression-mapping.sh | 8 ++++++-- sbin/es-repo-mapping.sh | 8 ++++++-- 2 files changed, 12 insertions(+), 4 deletions(-) diff --git a/sbin/es-regression-mapping.sh b/sbin/es-regression-mapping.sh index 5ae0f71..9d01bf9 100755 --- a/sbin/es-regression-mapping.sh +++ b/sbin/es-regression-mapping.sh @@ -1,14 +1,18 @@ #!/bin/sh +. $CCI_SRC/container/defconfig.sh + +load_service_authentication + # check whether regression index has created -status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) +status_code=$(curl -sSIL -u "${ES_USER}:${ES_PASSWORD}" -w "%{http_code}\\n" -o /dev/null http://localhost:9200/regression) if [ "$status_code" -eq 200 ] then echo "regression index has been created, exit." else echo "begin create index." - curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/regression' -u "${ES_USER}:${ES_PASSWORD}" -d '{ "mappings": { "dynamic": false, "properties": { diff --git a/sbin/es-repo-mapping.sh b/sbin/es-repo-mapping.sh index 290efb3..368dc94 100755 --- a/sbin/es-repo-mapping.sh +++ b/sbin/es-repo-mapping.sh @@ -2,13 +2,17 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/container/defconfig.sh + +load_service_authentication + # Check if "repo" index already exists. -status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) +status_code=$(curl -sSIL -u "${ES_USER}:${ES_PASSWORD}" -w "%{http_code}\n" -o /dev/null http://localhost:9200/repo) [ $status_code -eq 200 ] && echo '"repo" index already exists.' && exit # Create "repo" index. echo 'Start to create "repo" index.' -curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -d ' +curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/repo' -u "${ES_USER}:${ES_PASSWORD}" -d ' { "mappings": { "properties": { -- 2.23.0

1 0

[PATCH v5 lkp-tests 2/2] lib/rpm.sh: add add_repo function
by Wang Yong 27 Apr '21

27 Apr '21

add function for rpmbuild-pkg and install-rpm Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/rpm.sh | 20 ++++++++++++++++++++ tests/install-rpm | 22 +--------------------- tests/rpmbuild-pkg | 6 ++++++ 3 files changed, 27 insertions(+), 21 deletions(-) create mode 100755 lib/rpm.sh diff --git a/lib/rpm.sh b/lib/rpm.sh new file mode 100755 index 000000000..42bc7f5e0 --- /dev/null +++ b/lib/rpm.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<-EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" + [${custom_repo_name[$i]}] + name=${custom_repo_name[$i]} + baseurl=${custom_repo_addr[$i]} + enabled=1 + gpgcheck=0 + priority=100 + + EOF + done +} diff --git a/tests/install-rpm b/tests/install-rpm index 13beecbee..7449b3bd1 100755 --- a/tests/install-rpm +++ b/tests/install-rpm @@ -9,6 +9,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/log.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" : "${DISTRO:=openeuler}" @@ -19,29 +20,8 @@ [ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" [ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" -custom_repo_name=($custom_repo_name) -custom_repo_addr=($custom_repo_addr) rpm_name=($rpm_name) -add_repo() -{ - custom_repo_name=($custom_repo_name) - custom_repo_addr=($custom_repo_addr) - - for i in ${!custom_repo_name[@]} - do - cat <<-EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" - [${custom_repo_name[$i]}] - name=${custom_repo_name[$i]} - baseurl=${custom_repo_addr[$i]} - enabled=1 - gpgcheck=0 - priority=100 - - EOF - done -} - install_rpm() { log_info "Starting install test: yum install -y ${rpm_name[@]}" diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 312582d16..5c694b9d4 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -3,10 +3,15 @@ # - compat_os # - repo_name # - repo_addr +# - custom_repo_name +# - custom_repo_addr # Git repo jobs only use the first two parameters, SRPMs jobs use all +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" @@ -133,6 +138,7 @@ update_repo_mq() -d "{\"upload_rpms\": ["${full_list}"]}" } +add_repo [ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg -- 2.23.0

1 0

[PATCH v5 lkp-tests 1/2] tests/install-rpm: add suite install-rpm
by Wang Yong 27 Apr '21

27 Apr '21

add install-rpm to install/remove and ldd test for rpm packages Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/install-rpm | 1 + jobs/install-rpm.yaml | 4 ++ tests/install-rpm | 83 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 88 insertions(+) create mode 100644 distro/depends/install-rpm create mode 100755 jobs/install-rpm.yaml create mode 100755 tests/install-rpm diff --git a/distro/depends/install-rpm b/distro/depends/install-rpm new file mode 100644 index 000000000..602479c9a --- /dev/null +++ b/distro/depends/install-rpm @@ -0,0 +1 @@ +dnf-plugins-core diff --git a/jobs/install-rpm.yaml b/jobs/install-rpm.yaml new file mode 100755 index 000000000..9afa9d992 --- /dev/null +++ b/jobs/install-rpm.yaml @@ -0,0 +1,4 @@ +suite: install-rpm +category: functional + +install-rpm: diff --git a/tests/install-rpm b/tests/install-rpm new file mode 100755 index 000000000..13beecbee --- /dev/null +++ b/tests/install-rpm @@ -0,0 +1,83 @@ +#!/bin/bash +# - compat_os +# - rpm_name +# - custom_repo_name +# - custom_repo_addr +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/log.sh + +: "${compat_os:=budding-openeuler}" +: "${DISTRO:=openeuler}" +: "${basearch:=$(arch)}" +. $LKP_SRC/distro/${DISTRO} + +[ -n "${rpm_name}" ] || die "rpm_name is empty" +[ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" +[ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" + +custom_repo_name=($custom_repo_name) +custom_repo_addr=($custom_repo_addr) +rpm_name=($rpm_name) + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<-EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" + [${custom_repo_name[$i]}] + name=${custom_repo_name[$i]} + baseurl=${custom_repo_addr[$i]} + enabled=1 + gpgcheck=0 + priority=100 + + EOF + done +} + +install_rpm() +{ + log_info "Starting install test: yum install -y ${rpm_name[@]}" + yum install -y ${rpm_name[@]} + + return 0 +} + +ldd_test() +{ + log_info "Starting ldd test" + for i in $(rpm -qa ${rpm_name[@]}) + do + ldd_list=$(rpm -ql $i | awk -F'/' '$3 ~ /bin|sbin|lib|lib64/') + [ -z "${ldd_list}" ] || { + for i in ${ldd_list[@]} + do + [[ -f "$i" && -x "$i" ]] && { + echo "ldd for $i" + ldd -d -r $i + } + done + } + done +} + +remove_rpm() +{ + log_info "Starting remove test: yum remove -y ${rpm_name[@]}" + yum remove -y ${rpm_name[@]} + + return 0 +} + +distro_install_depends install-rpm +add_repo +install_rpm +ldd_test +remove_rpm -- 2.23.0

1 0

[PATCH v5 lkp-tests] tests/qcow2rootfs-x86_64: fix failed to build container using absolute path
by Wang Chenglong 27 Apr '21

27 Apr '21

[Error] root@2288hv5-2s44p-384g--b6 /c/lkp-tests/tests# /c/compass-ci/container/qcow2rootfs/build /c/compass-ci/container/qcow2rootfs/build: line 5: ../defconfig.sh: No such file or directory /c/compass-ci/container/qcow2rootfs/build: line 7: docker_skip_rebuild: command not found Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- tests/qcow2rootfs-x86_64 | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/tests/qcow2rootfs-x86_64 b/tests/qcow2rootfs-x86_64 index f698fb7ec..a783dc1a6 100755 --- a/tests/qcow2rootfs-x86_64 +++ b/tests/qcow2rootfs-x86_64 @@ -9,12 +9,17 @@ set_qcow2rootfs_tools() { export QCOW2_PATH="/c/compass-ci/container/qcow2rootfs" - git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci || return 1 + export CCI_SRC="/c/compass-ci" + git clone https://gitee.com/wu_fengguang/compass-ci.git /c/compass-ci rm ${QCOW2_PATH}/root/etc/yum.repos.d/* wget -O ${QCOW2_PATH}/root/etc/yum.repos.d/CentOS-Base.repo https://repo.huaweicloud.com/repository/conf/CentOS-7-reg.repo - ${QCOW2_PATH}/build + cd ${QCOW2_PATH} + ./build || { + echo "build container failed~" + exit 1 + } sed -i 's#$EXTRACT_ROOT/bin/extract $QCOW2_FILE $tar_out_file#$EXTRACT_ROOT/bin/extract $EXTRACT_ROOT/qcow2-dir/$QCOW2_NAME $tar_out_file#' \ - ${QCOW2_PATH}/bin/mail + ${QCOW2_PATH}/bin/main } @@ -52,7 +57,13 @@ extract_rootfs() export ROOTFS_DIR="${qcow2_version}-$(date +%Y-%m-%d-%H-%M-%S)" export ROOTFS_PATH="${MOUNT_POINT}/openeuler/${qcow2_arch}/automatic-build/${ROOTFS_DIR}" mkdir -p $ROOTFS_PATH - ${QCOW2_PATH}/run ${IMG_PATH}/${QCOW2_NAME} ${ROOTFS_PATH} + cd $CCI_SRC/container/dracut-initrd + ./build || { + echo "build container failed~" + exit 1 + } + cd ${QCOW2_PATH} + ./run ${IMG_PATH}/${QCOW2_NAME} ${ROOTFS_PATH} } set_resolv_config() -- 2.23.0

1 0

[PATCH v4 lkp-tests] lib/bootstrap.sh: fix the output error when ping not exist
by Wang Chenglong 27 Apr '21

27 Apr '21

[Why] If the ping command exists, then run test_ping, if not, then skip. [Error] [ 179.335889] Kernel tests: Boot OK! [ 180.180309] LKP: waiting for network... [ 182.991732] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found [ 183.010815] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 070dabf46..58f0531a7 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -88,6 +88,10 @@ net_devices_link() test_ping() { + has_cmd ping || { + echo "command not found: ping, skip ping test." + return 0 + } ping -c 1 -w 1 $LKP_SERVER > /dev/null } -- 2.23.0

1 0

[PATCH compass-ci] lib/compare_error_messages.rb: fix filename matching rule
by Lin Jiaxin 27 Apr '21

27 Apr '21

Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/compare_error_messages.rb | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/lib/compare_error_messages.rb b/lib/compare_error_messages.rb index 93b38ed..7c53580 100644 --- a/lib/compare_error_messages.rb +++ b/lib/compare_error_messages.rb @@ -37,7 +37,9 @@ module CEM error_lines = ErrorMessages.new(previous_result_file).obtain_error_messages_by_error_id(error_id, true) error_lines.each do |error_line| # "src/ssl_sock.c:1454:104: warning: unused parameter 'al' [-Wunused-parameter]" => "src/ssl_sock" - filenames << $1.chomp(File.extname($1)) if error_line =~ /(.*)(:\d+){2}: (error|warning):/ + # "/tmp/lkp/bart-git/src/bart/src/num/lapack.c:20:10: fatal error: lapacke.h: No such file or directory" + # => "/tmp/lkp/bart-git/src/bart/src/num/lapack" + filenames << $1.chomp(File.extname($1)) if error_line =~ /(.*)(:\d+){2}: ((fatal )?error|warning):/ end File.open(later_result_file).each_line do |line| -- 2.23.0

1 0

[PATCH compass-ci] assign-account/start: delete useless code
by Wu Zhende 27 Apr '21

27 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/assign-account/start | 2 -- 1 file changed, 2 deletions(-) diff --git a/container/assign-account/start b/container/assign-account/start index 42b34c2..b926853 100755 --- a/container/assign-account/start +++ b/container/assign-account/start @@ -4,8 +4,6 @@ . ../defconfig.sh -load_service_authentication - docker_rm assign-account cmd=( -- 2.23.0

1 0

[PATCH compass-ci] container/defconfig: fix file does not exist
by Wu Zhende 27 Apr '21

27 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/defconfig.rb | 5 ++++- container/defconfig.sh | 4 +++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/container/defconfig.rb b/container/defconfig.rb index 0a8e4d5..ba0e230 100755 --- a/container/defconfig.rb +++ b/container/defconfig.rb @@ -23,7 +23,10 @@ def relevant_defaults(names) end def relevant_service_authentication(names) - hash = YAML.load_file('/etc/compass-ci/passwd.yaml') || {} + file_name = '/etc/compass-ci/passwd.yaml' + return {} unless File.exist?(file_name) + + hash = YAML.load_file(file_name) || {} hash.select { |k, _| names.include? k } end diff --git a/container/defconfig.sh b/container/defconfig.sh index 64a434f..c217d5f 100755 --- a/container/defconfig.sh +++ b/container/defconfig.sh @@ -23,7 +23,9 @@ load_cci_defaults() load_service_authentication() { shopt -s nullglob - create_yaml_variables '/etc/compass-ci/passwd.yaml' + file_name='/etc/compass-ci/passwd.yaml' + [ -f $file_name ] || return + create_yaml_variables $file_name } docker_rm() -- 2.23.0

1 0

[PATCH v4 lkp-tests] lib/bootstrap.sh: fix the output error when ping not exist
by Wang Chenglong 27 Apr '21

27 Apr '21

[Why] If the ping command exists, then run test_ping, if not, then skip. [Error] [ 179.335889] Kernel tests: Boot OK! [ 180.180309] LKP: waiting for network... [ 182.991732] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found [ 183.010815] /lkp/lkp/src/lib/bootstrap.sh: line 90: ping: command not found Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- lib/bootstrap.sh | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 070dabf46..58f0531a7 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -88,6 +88,10 @@ net_devices_link() test_ping() { + has_cmd ping || { + echo "command not found: ping, skip ping test." + return 0 + } ping -c 1 -w 1 $LKP_SERVER > /dev/null } -- 2.23.0

2 2

[PATCH v4 lkp-tests 2/2] lib/rpm.sh: add add_repo function
by Wang Yong 27 Apr '21

27 Apr '21

add function for rpmbuild-pkg and install-rpm Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/rpm.sh | 20 ++++++++++++++++++++ tests/install-rpm | 22 +--------------------- tests/rpmbuild-pkg | 6 ++++++ 3 files changed, 27 insertions(+), 21 deletions(-) create mode 100755 lib/rpm.sh diff --git a/lib/rpm.sh b/lib/rpm.sh new file mode 100755 index 000000000..4c4cbc078 --- /dev/null +++ b/lib/rpm.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<-EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" + [${custom_repo_name[$i]}] + name=${custom_repo_name[$i]} + baseurl=${custom_repo_addr[$i]} + enabled=1 + gpgcheck=0 + priority=100 + + EOF +done +} diff --git a/tests/install-rpm b/tests/install-rpm index d69d2d4c1..7449b3bd1 100755 --- a/tests/install-rpm +++ b/tests/install-rpm @@ -9,6 +9,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/log.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" : "${DISTRO:=openeuler}" @@ -19,29 +20,8 @@ [ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" [ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" -custom_repo_name=($custom_repo_name) -custom_repo_addr=($custom_repo_addr) rpm_name=($rpm_name) -add_repo() -{ - custom_repo_name=($custom_repo_name) - custom_repo_addr=($custom_repo_addr) - - for i in ${!custom_repo_name[@]} - do - cat <<EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" -[${custom_repo_name[$i]}] -name=${custom_repo_name[$i]} -baseurl=${custom_repo_addr[$i]} -enabled=1 -gpgcheck=0 -priority=100 - -EOF - done -} - install_rpm() { log_info "Starting install test: yum install -y ${rpm_name[@]}" diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 312582d16..5c694b9d4 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -3,10 +3,15 @@ # - compat_os # - repo_name # - repo_addr +# - custom_repo_name +# - custom_repo_addr # Git repo jobs only use the first two parameters, SRPMs jobs use all +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" @@ -133,6 +138,7 @@ update_repo_mq() -d "{\"upload_rpms\": ["${full_list}"]}" } +add_repo [ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH v4 lkp-tests 1/2] tests/install-rpm: add suite install-rpm
by Wang Yong 27 Apr '21

27 Apr '21

add install-rpm to install/remove and ldd test for rpm packages Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/install-rpm | 1 + jobs/install-rpm.yaml | 4 ++ tests/install-rpm | 83 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 88 insertions(+) create mode 100644 distro/depends/install-rpm create mode 100755 jobs/install-rpm.yaml create mode 100755 tests/install-rpm diff --git a/distro/depends/install-rpm b/distro/depends/install-rpm new file mode 100644 index 000000000..602479c9a --- /dev/null +++ b/distro/depends/install-rpm @@ -0,0 +1 @@ +dnf-plugins-core diff --git a/jobs/install-rpm.yaml b/jobs/install-rpm.yaml new file mode 100755 index 000000000..9afa9d992 --- /dev/null +++ b/jobs/install-rpm.yaml @@ -0,0 +1,4 @@ +suite: install-rpm +category: functional + +install-rpm: diff --git a/tests/install-rpm b/tests/install-rpm new file mode 100755 index 000000000..d69d2d4c1 --- /dev/null +++ b/tests/install-rpm @@ -0,0 +1,83 @@ +#!/bin/bash +# - compat_os +# - rpm_name +# - custom_repo_name +# - custom_repo_addr +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/log.sh + +: "${compat_os:=budding-openeuler}" +: "${DISTRO:=openeuler}" +: "${basearch:=$(arch)}" +. $LKP_SRC/distro/${DISTRO} + +[ -n "${rpm_name}" ] || die "rpm_name is empty" +[ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" +[ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" + +custom_repo_name=($custom_repo_name) +custom_repo_addr=($custom_repo_addr) +rpm_name=($rpm_name) + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" +[${custom_repo_name[$i]}] +name=${custom_repo_name[$i]} +baseurl=${custom_repo_addr[$i]} +enabled=1 +gpgcheck=0 +priority=100 + +EOF + done +} + +install_rpm() +{ + log_info "Starting install test: yum install -y ${rpm_name[@]}" + yum install -y ${rpm_name[@]} + + return 0 +} + +ldd_test() +{ + log_info "Starting ldd test" + for i in $(rpm -qa ${rpm_name[@]}) + do + ldd_list=$(rpm -ql $i | awk -F'/' '$3 ~ /bin|sbin|lib|lib64/') + [ -z "${ldd_list}" ] || { + for i in ${ldd_list[@]} + do + [[ -f "$i" && -x "$i" ]] && { + echo "ldd for $i" + ldd -d -r $i + } + done + } + done +} + +remove_rpm() +{ + log_info "Starting remove test: yum remove -y ${rpm_name[@]}" + yum remove -y ${rpm_name[@]} + + return 0 +} + +distro_install_depends install-rpm +add_repo +install_rpm +ldd_test +remove_rpm -- 2.23.0

2 2

[PATCH v2 compass-ci] sbin: configuring ES authentication information
by Wu Zhende 26 Apr '21

26 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-accounts-mapping.sh | 8 ++++++-- sbin/es-jobs-mapping.sh | 10 +++++++--- 2 files changed, 13 insertions(+), 5 deletions(-) diff --git a/sbin/es-accounts-mapping.sh b/sbin/es-accounts-mapping.sh index c4ac2fc..580b459 100755 --- a/sbin/es-accounts-mapping.sh +++ b/sbin/es-accounts-mapping.sh @@ -2,15 +2,19 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/container/defconfig.sh + +load_service_authentication + # check whether accounts index has created -status_code=$(curl -sSIL -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) +status_code=$(curl -sSIL -u "${ES_USER}:${ES_PASSWORD}" -w "%{http_code}\\n" -o /dev/null http://localhost:9200/accounts) if [ "$status_code" -eq 200 ] then echo "accounts index has been created, exit." else echo "begin create index." - curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/accounts' -u "${ES_USER}:${ES_PASSWORD}" -d '{ "mappings": { "dynamic": false, "properties": { diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 67b1ed1..15000d9 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -2,6 +2,10 @@ # SPDX-License-Identifier: MulanPSL-2.0+ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +. $CCI_SRC/container/defconfig.sh + +load_service_authentication + # Determine whether curl is installed. If not, install curl. if ! [ -x "$(command -v curl)" ] then @@ -28,14 +32,14 @@ then fi # Determine whether jobs index has created -status_code=$(curl -sSIL -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) +status_code=$(curl -sSIL -u "${ES_USER}:${ES_PASSWORD}" -w "%{http_code}\n" -o /dev/null http://localhost:9200/jobs) if [ $status_code -eq 200 ] then echo "jobs index has create, exit." else echo "jobs index not exists, begin create index." - curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -d '{ + curl -sSH 'Content-Type: Application/json' -XPUT 'http://localhost:9200/jobs' -u "${ES_USER}:${ES_PASSWORD}" -d '{ "mappings": { "dynamic": false, "dynamic_templates": [ @@ -185,7 +189,7 @@ else echo "create jobs index failed." else echo "set index.mapping.total_fields.limit: 10000" - curl -sS -XPUT 127.0.0.1:9200/jobs/_settings -H 'Content-Type: application/json' \ + curl -sS -XPUT 127.0.0.1:9200/jobs/_settings -u "${ES_USER}:${ES_PASSWORD}" -H 'Content-Type: application/json' \ -d '{"index.mapping.total_fields.limit": 10000}' fi fi -- 2.23.0

2 1

[PATCH compass-ci] container/openresty-proxy-cache: fix failed to git clone
by Liu Yinsi 26 Apr '21

26 Apr '21

[error] /c/compass-ci/container/openresty-proxy-cache# ./build Cloning into 'openresty'... fatal: remote error: access denied or repository not exported: /gitee.com/cuiyili/openresty.git [why] lkp-tests/tests/deploy-ci: ... cat >> /etc/gitconfig <<-EOF [url "git://$GIT_SERVER/gitee.com"] insteadOf=https://gitee.com ... /etc/gitconfig cause failed to git clone https://gitee.com/cuiyili/openresty.git, https -> http to fix this problem. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/openresty-proxy-cache/build | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/openresty-proxy-cache/build b/container/openresty-proxy-cache/build index 76652097..3635507f 100755 --- a/container/openresty-proxy-cache/build +++ b/container/openresty-proxy-cache/build @@ -8,6 +8,6 @@ docker_skip_rebuild "openresty:proxy_cache" [ -d "openresty" ] && rm -rf openresty -git clone https://gitee.com/cuiyili/openresty.git || exit 1 +git clone http://gitee.com/cuiyili/openresty.git || exit 1 docker build -t openresty:proxy_cache . -f openresty/Dockerfile -- 2.23.0

2 2

[PATCH compass-ci] container/srv-http: allow user download deps files
by Liu Yinsi 26 Apr '21

26 Apr '21

[why] run-ipconfig.cgz is necessary for user locally deploy compass-ci, when they want to run a qemu, so allow /srv/initrd/deps for download. See [PATCH v5 compass-ci] sbin/download-rootfs: download osimage { ... wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" ... } Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- container/srv-http/start-initrd | 1 + 1 file changed, 1 insertion(+) diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd index 457465b..41958da 100755 --- a/container/srv-http/start-initrd +++ b/container/srv-http/start-initrd @@ -15,6 +15,7 @@ cmd=( -v /srv/initrd/dockerimage:/srv/initrd/dockerimage:ro -v /srv/initrd/rpmbuild-pkg:/srv/initrd/rpmbuild-pkg:ro -v /srv/initrd/osimage:/srv/initrd/osimage:ro + -v /srv/initrd/deps:/srv/initrd/deps:ro -v /etc/localtime:/etc/localtime:ro -d srv-http -- 2.23.0

1 0

[PATCH compass-ci] container/mail-robot: fix bug
by Luan Shengde 26 Apr '21

26 Apr '21

add init variable: err_message when execute line: err_message += <<~EMAIL_MESSAGE it may throw error: ERROR -- : exception while processing events: undefined method `+' for nil:NilClass Backtrace: Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/mail-robot/lib/assign-account-fail-eamil.rb | 1 + 1 file changed, 1 insertion(+) diff --git a/container/mail-robot/lib/assign-account-fail-eamil.rb b/container/mail-robot/lib/assign-account-fail-eamil.rb index 31692358..a7f43f11 100755 --- a/container/mail-robot/lib/assign-account-fail-eamil.rb +++ b/container/mail-robot/lib/assign-account-fail-eamil.rb @@ -4,6 +4,7 @@ # frozen_string_literal: true def email_err_message(message) + err_message = '' case message when 'NO_COMMIT_URL' err_message = <<~EMAIL_MESSAGE -- 2.23.0

1 0

[PATCH compass-ci] sbin/download-rootfs: optimise error msg
by Liu Yinsi 26 Apr '21

26 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sbin/download-rootfs b/sbin/download-rootfs index 4edd1952..0d0d1c7b 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -27,7 +27,7 @@ wget_download() { local download_url=$1 wget $download_url || { - log_error "failed to download os rootfs: $download_url" + log_error "failed to download file: $download_url" exit 1 } } -- 2.23.0

1 0

[PATCH v5 compass-ci] sbin/download-rootfs: download osimage
by Liu Yinsi 26 Apr '21

26 Apr '21

the default os_mount=initramfs, so download osimage to run qemu kvm. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sbin/download-rootfs | 41 +++++++++++++++++++++++++++-------------- 1 file changed, 27 insertions(+), 14 deletions(-) diff --git a/sbin/download-rootfs b/sbin/download-rootfs index 68a68039..4edd1952 100755 --- a/sbin/download-rootfs +++ b/sbin/download-rootfs @@ -18,23 +18,36 @@ os_version=${3:-20.03} load_cci_defaults -rootfs_dir=/srv/os/${os}/${os_arch}/${os_version} +rootfs_dir="/srv/os/${os}/${os_arch}/${os_version}" +osimage_dir="/srv/initrd/osimage/${os}/${os_arch}/${os_version}" -download_os_rootfs() -{ - mkdir -p $rootfs_dir - cd $rootfs_dir +mkdir -p $rootfs_dir $osimage_dir - download_url="http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" - wget $download_url +wget_download() +{ + local download_url=$1 + wget $download_url || { + log_error "failed to download os rootfs: $download_url" + exit 1 + } +} - [ "$?" -ne 0 ] && { - log_error "failed to download os rootfs: $download_url" - exit 1 - } +download_os_rootfs() +{ + cd $rootfs_dir && { + wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" + + gzip -dc ${os_version}.cgz | cpio -id + rm -rf ${os_version}.cgz + } +} - gzip -dc ${os_version}.cgz | cpio -id - rm -rf ${os_version}.cgz +download_osimage() +{ + cd $osimage_dir && { + wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage/…" + wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" + } } is_virtual() @@ -42,4 +55,4 @@ is_virtual() dmidecode -s system-product-name | grep -iq "virtual" } -is_virtual || download_os_rootfs +is_virtual || ( download_os_rootfs && download_osimage ) -- 2.23.0

2 2

[PATCH compass-ci] delimiter: delete unused require
by Cao Xueliang 26 Apr '21

26 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/git_bisect.rb | 1 - src/delimiter/utils.rb | 1 - 2 files changed, 2 deletions(-) diff --git a/lib/git_bisect.rb b/lib/git_bisect.rb index c0c0a60..eaaee78 100644 --- a/lib/git_bisect.rb +++ b/lib/git_bisect.rb @@ -4,7 +4,6 @@ require 'json' require_relative 'error_messages' -require_relative 'sched_client' require_relative '../src/delimiter/utils' require_relative "#{ENV['LKP_SRC']}/lib/monitor" diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index 5ecbeb5..79bb658 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -8,7 +8,6 @@ require 'yaml' require 'fileutils' require_relative './constants' -require_relative '../../lib/sched_client' require_relative '../../lib/assist_result_client' require_relative '../../lib/compare_error_messages' require_relative "#{ENV['LKP_SRC']}/lib/monitor" -- 2.23.0

1 0

[PATCH v3 lkp-tests] lib/rpm.sh: add add_repo function
by Wang Yong 26 Apr '21

26 Apr '21

add function for rpmbuild-pkg and install-rpm Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/rpm.sh | 20 ++++++++++++++++++++ tests/install-rpm | 22 +--------------------- tests/rpmbuild-pkg | 6 ++++++ 3 files changed, 27 insertions(+), 21 deletions(-) create mode 100755 lib/rpm.sh diff --git a/lib/rpm.sh b/lib/rpm.sh new file mode 100755 index 000000000..4c4cbc078 --- /dev/null +++ b/lib/rpm.sh @@ -0,0 +1,20 @@ +#!/bin/bash + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<-EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" + [${custom_repo_name[$i]}] + name=${custom_repo_name[$i]} + baseurl=${custom_repo_addr[$i]} + enabled=1 + gpgcheck=0 + priority=100 + + EOF +done +} diff --git a/tests/install-rpm b/tests/install-rpm index d69d2d4c1..7449b3bd1 100755 --- a/tests/install-rpm +++ b/tests/install-rpm @@ -9,6 +9,7 @@ . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh . $LKP_SRC/lib/log.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" : "${DISTRO:=openeuler}" @@ -19,29 +20,8 @@ [ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" [ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" -custom_repo_name=($custom_repo_name) -custom_repo_addr=($custom_repo_addr) rpm_name=($rpm_name) -add_repo() -{ - custom_repo_name=($custom_repo_name) - custom_repo_addr=($custom_repo_addr) - - for i in ${!custom_repo_name[@]} - do - cat <<EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" -[${custom_repo_name[$i]}] -name=${custom_repo_name[$i]} -baseurl=${custom_repo_addr[$i]} -enabled=1 -gpgcheck=0 -priority=100 - -EOF - done -} - install_rpm() { log_info "Starting install test: yum install -y ${rpm_name[@]}" diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index 312582d16..5c694b9d4 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -3,10 +3,15 @@ # - compat_os # - repo_name # - repo_addr +# - custom_repo_name +# - custom_repo_addr # Git repo jobs only use the first two parameters, SRPMs jobs use all +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/rpm.sh : "${compat_os:=budding-openeuler}" @@ -133,6 +138,7 @@ update_repo_mq() -d "{\"upload_rpms\": ["${full_list}"]}" } +add_repo [ -n "$repo_name" ] && from_srpm || from_git build_rpm upload_rpm_pkg -- 2.23.0

1 1

[PATCH v2 lkp-tests 1/2] tests/install-rpm: add suite install-rpm
by Wang Yong 26 Apr '21

26 Apr '21

add install-rpm to install/remove and ldd test for rpm packages Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- distro/depends/install-rpm | 1 + jobs/install-rpm.yaml | 4 ++ tests/install-rpm | 83 ++++++++++++++++++++++++++++++++++++++ 3 files changed, 88 insertions(+) create mode 100644 distro/depends/install-rpm create mode 100755 jobs/install-rpm.yaml create mode 100755 tests/install-rpm diff --git a/distro/depends/install-rpm b/distro/depends/install-rpm new file mode 100644 index 000000000..602479c9a --- /dev/null +++ b/distro/depends/install-rpm @@ -0,0 +1 @@ +dnf-plugins-core diff --git a/jobs/install-rpm.yaml b/jobs/install-rpm.yaml new file mode 100755 index 000000000..9afa9d992 --- /dev/null +++ b/jobs/install-rpm.yaml @@ -0,0 +1,4 @@ +suite: install-rpm +category: functional + +install-rpm: diff --git a/tests/install-rpm b/tests/install-rpm new file mode 100755 index 000000000..d69d2d4c1 --- /dev/null +++ b/tests/install-rpm @@ -0,0 +1,83 @@ +#!/bin/bash +# - compat_os +# - rpm_name +# - custom_repo_name +# - custom_repo_addr +# user can add custom_repo_name and custom_repo_addr to setup +# local repo, value can be an array + +. $LKP_SRC/lib/debug.sh +. $LKP_SRC/lib/upload.sh +. $LKP_SRC/lib/log.sh + +: "${compat_os:=budding-openeuler}" +: "${DISTRO:=openeuler}" +: "${basearch:=$(arch)}" +. $LKP_SRC/distro/${DISTRO} + +[ -n "${rpm_name}" ] || die "rpm_name is empty" +[ -n "${custom_repo_name}" ] || die "custom_repo_name is empty" +[ -n "${custom_repo_addr}" ] || die "custom_repo_addr is empty" + +custom_repo_name=($custom_repo_name) +custom_repo_addr=($custom_repo_addr) +rpm_name=($rpm_name) + +add_repo() +{ + custom_repo_name=($custom_repo_name) + custom_repo_addr=($custom_repo_addr) + + for i in ${!custom_repo_name[@]} + do + cat <<EOF >> /etc/yum.repos.d/"${custom_repo_name[$i]}.repo" +[${custom_repo_name[$i]}] +name=${custom_repo_name[$i]} +baseurl=${custom_repo_addr[$i]} +enabled=1 +gpgcheck=0 +priority=100 + +EOF + done +} + +install_rpm() +{ + log_info "Starting install test: yum install -y ${rpm_name[@]}" + yum install -y ${rpm_name[@]} + + return 0 +} + +ldd_test() +{ + log_info "Starting ldd test" + for i in $(rpm -qa ${rpm_name[@]}) + do + ldd_list=$(rpm -ql $i | awk -F'/' '$3 ~ /bin|sbin|lib|lib64/') + [ -z "${ldd_list}" ] || { + for i in ${ldd_list[@]} + do + [[ -f "$i" && -x "$i" ]] && { + echo "ldd for $i" + ldd -d -r $i + } + done + } + done +} + +remove_rpm() +{ + log_info "Starting remove test: yum remove -y ${rpm_name[@]}" + yum remove -y ${rpm_name[@]} + + return 0 +} + +distro_install_depends install-rpm +add_repo +install_rpm +ldd_test +remove_rpm -- 2.23.0

2 2

[PATCH compass-ci] extract-stats: fix no new_error_id
by Cao Xueliang 26 Apr '21

26 Apr '21

regression_client need add auth info. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/extract-stats/regression_client.cr | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/extract-stats/regression_client.cr b/src/extract-stats/regression_client.cr index 586ea20..4ff8ab7 100644 --- a/src/extract-stats/regression_client.cr +++ b/src/extract-stats/regression_client.cr @@ -5,7 +5,13 @@ class RegressionClient HOST = (ENV.has_key?("ES_HOST") ? ENV["ES_HOST"] : JOB_ES_HOST) PORT = (ENV.has_key?("ES_PORT") ? ENV["ES_PORT"] : JOB_ES_PORT).to_i32 - def initialize(host = HOST, port = PORT) + def initialize(host = HOST, port = PORT, auth = true) + if auth + user = ENV["ES_USER"]? + password = ENV["ES_PASSWORD"]? + host = "#{user}:#{URI.encode_www_form(password)}@#{host}" if user && password + end + @client = Elasticsearch::API::Client.new({:host => host, :port => port}) end -- 2.23.0

1 0

[PATCH v2 lkp-tests] etc/latency: added
by Lu Weitao 26 Apr '21

26 Apr '21

[Why] compass-ci/lib/compare_matrixes.rb need lkp-test/etc/latency Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- etc/latency | 13 +++++++++++++ 1 file changed, 13 insertions(+) create mode 100644 etc/latency diff --git a/etc/latency b/etc/latency new file mode 100644 index 000000000..25dab098e --- /dev/null +++ b/etc/latency @@ -0,0 +1,13 @@ +dbench.max_latency +dd.kill_time +dd.startup_time +ebizzy.throughput.per_thread.stddev_percent +fileio.latency_95th_ms +fileio.latency_max_ms +iostat\..*\.(await|r_await|w_await|svctm|avgqu-sz)\.max +latency_stats\..* +oltp.latency_95th_ms +oltp.latency_max_ms +oltp.latency_min_ms +tbench.max_latency +vm-scalability.stddev -- 2.23.0

1 0