- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] sparrow/3-code/dev-env: set default value
by Liu Yinsi 20 Apr '21

20 Apr '21

[error] submit_id=e628930d-f8ce-4ea1-b323-f8593c6f60cf 2021-04-19 20:59:08 +0800 WARN -- skip non-executable /c/lkp-tests/tests/self-test submit /c/lkp-tests/jobs/host-info.yaml failed, got job id=0, error: The /srv/initrd/lkp/latest/lkp-aarch64.cgz does not exist. [why] when selftest, if not to define $lkp_initrd_user in /etc/compass-ci/defaults/*.yaml, when submit job, it will not to find /srv/initrd/lkp/$lkp_initrd_user/lkp-$arch.cgz. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 969889ed..e02de355 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -55,12 +55,14 @@ server_name=$(hostname | cut -f1 -d.) : ${MAILDIR:=/srv/cci/Maildir/.compass-ci} : ${SENT_MAILDIR:=/srv/cci/Maildir/.sent} : ${NETDATA_PORT:=20013} +: ${lkp_initrd_user:=latest} mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF SCHED_HOST: $sched_host SCHED_PORT: $sched_port lab: $lab +lkp_initrd_user: $lkp_initrd_user EOF [[ -f /etc/compass-ci/defaults/upstream-config ]] || { -- 2.23.0

1 0

[PATCH compass-ci] sparrow/1-storage/permission: set lkp owner and group
by Liu Yinsi 20 Apr '21

20 Apr '21

/srv/initrd/deps and /srv/initrd/pkg permission must be lkp:lkp, or build pkg and deps are unable to transfer to the directories. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index db33a93..3c229c5 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -13,6 +13,9 @@ chmod_dir=( /srv/rabbitmq /c/git-repos /srv/tmp/public + /srv/initrd + /srv/initrd/pkg + /srv/initrd/deps ) chmod 775 "${chmod_dir[@]}" @@ -20,6 +23,8 @@ chmod 775 "${chmod_dir[@]}" chown_dir=( /srv/result /srv/initrd + /srv/initrd/pkg + /srv/initrd/deps /srv/es /srv/cache/netdata_cache /srv/cache/netdata_lib -- 2.23.0

1 0

[PATCH lkp-tests] sbin/submit: support nr_run
by Li Yuanchao 20 Apr '21

20 Apr '21

If nr_run is a integer and nr_run > 1, submit nr_run times. Else, submit once. [usage] submit xxx xxx nr_run=3 xxx.yaml Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/submit | 25 +++++++++++++++---------- 1 file changed, 15 insertions(+), 10 deletions(-) diff --git a/sbin/submit b/sbin/submit index edb207cc0..2e6ad8960 100755 --- a/sbin/submit +++ b/sbin/submit @@ -195,16 +195,21 @@ ARGV.each do |jobfile| scheduler_client = SchedulerClient.new(job['SCHED_HOST'], job['SCHED_PORT']) # submit job - job_json = job_hash.to_json - messages = scheduler_client.submit_job(job_json) - JSON.parse(messages).each do |msg| - if msg['message'].empty? - result_roots << msg['result_root'] - job_ids << msg['job_id'].to_s - puts("submit #{jobfile}, got job id=#{msg['job_id']}") - else - opt_monitor = false - puts("submit #{jobfile} failed, got job id=#{msg['job_id']}, error: #{msg['message']}") + nr_run = job_hash['nr_run'] || 1 + nr_run = nr_run.to_i + nr_run = 1 if nr_run < 1 + nr_run.times do + job_json = job_hash.to_json + messages = scheduler_client.submit_job(job_json) + JSON.parse(messages).each do |msg| + if msg['message'].empty? + result_roots << msg['result_root'] + job_ids << msg['job_id'].to_s + puts("submit #{jobfile}, got job id=#{msg['job_id']}") + else + opt_monitor = false + puts("submit #{jobfile} failed, got job id=#{msg['job_id']}, error: #{msg['message']}") + end end end end -- 2.23.0

3 4

[PATCH lkp-tests 2/2] install-iso-firstboot: modify how job's IPXE response is retrieved
by Yu Chuan 19 Apr '21

19 Apr '21

[Why] Vmlinuz, initrd, reboot.cgz is all that is required for firstboot after ISO is installed. Therefore, the original os_mount=local is changed to custom_ipxe to obtain the ipxe response. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- jobs/install-iso-firstboot.yaml | 9 +-------- tests/install-iso-pre | 14 +++++++++----- 2 files changed, 10 insertions(+), 13 deletions(-) diff --git a/jobs/install-iso-firstboot.yaml b/jobs/install-iso-firstboot.yaml index 7be9eaaab184..c170b8a07f8a 100644 --- a/jobs/install-iso-firstboot.yaml +++ b/jobs/install-iso-firstboot.yaml @@ -1,10 +1,3 @@ suite: install-iso-firstboot -sleep: 1 - -# os_mount must be equal to 'local' -os_mount: local - -# kernel_custom_params must have 'boot_lv_suffix' and 'src_lv_suffix', -# and they must be the same. so can boot from the specify logical volume -kernel_custom_params: +custom_ipxe: diff --git a/tests/install-iso-pre b/tests/install-iso-pre index 768ffb2cca45..c89895a994cc 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -225,14 +225,18 @@ submit_install_iso_firstboot_yaml() local firstboot_yaml="${LKP_SRC}/jobs/install-iso-firstboot.yaml" [ -f "${firstboot_yaml}" ] || die "cannot find ${firstboot_yaml}" + custom_ipxe="#!ipxe\\\\n" + custom_ipxe="${custom_ipxe}\\\\ndhcp" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/initrd.lkp" + custom_ipxe="${custom_ipxe}\\\\ninitrd http://${INITRD_HTTP_HOST}:${INITRD_HTTP_PORT}/initrd/custom_bootstrap/reboot.cgz" + custom_ipxe="${custom_ipxe}\\\\nkernel http://${OS_HTTP_HOST}:${OS_HTTP_PORT}/os/${iso_os}/${iso_arch}/${iso_version}/boot/vmlinuz ip=dhcp root=/dev/mapper/os-${iso_os}_${iso_arch}_${iso_version}_${ROOTFS_SUFFIX} initrd=initrd.lkp initrd=reboot.cgz local rw" + custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" + + sed -i "/custom_ipxe:/ccustom_ipxe: \"${custom_ipxe}\"" $firstboot_yaml + ${LKP_SRC}/sbin/submit \ ${firstboot_yaml} \ suite=install-iso-firstboot-${ROOTFS_SUFFIX} \ - os=${iso_os} \ - os_arch=${iso_arch} \ - os_version=${iso_version} \ - os_mount=local \ - kernel_custom_params="src_lv_suffix=${ROOTFS_SUFFIX} boot_lv_suffix=${ROOTFS_SUFFIX}" \ testbox=$testbox || die "submit ${firstboot_yaml} failed" } -- 2.23.0

1 0

[PATCH lkp-tests 1/2] install-iso-pre: fix some bugs
by Yu Chuan 19 Apr '21

19 Apr '21

[What Bugs] 1. fix the use of secrets fields. 2. add mkfs.ext4 step after create logical volume. 3. vmlinux => vmlinuz 4. ks upload dir: - before: /srv/os/kickstart/ks/${iso_os}_${iso_arch}_${iso_version}.cfg - after: /srv/os/install/ks/${iso_os}_${iso_arch}_${iso_version}.ks 5. second-level hash problem when submit. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/install-iso-pre | 26 ++++++++++++++------------ 1 file changed, 14 insertions(+), 12 deletions(-) diff --git a/tests/install-iso-pre b/tests/install-iso-pre index ef4a0659fe02..768ffb2cca45 100755 --- a/tests/install-iso-pre +++ b/tests/install-iso-pre @@ -21,7 +21,7 @@ check_params() "HTTP_SERVER_HOST" "HTTP_SERVER_PORT" "HTTP_ROOT_DIR" - "ROOTFS_ROOT_PASSWD" + "secrets_ROOTFS_ROOT_PASSWD" "rootfs_disk" "RSYNC_SERVER_HOST" @@ -31,7 +31,7 @@ check_params() "NFS_ROOT_DIR" "LAB_Z9_URL" - "MY_TOKEN" + "secrets_MY_TOKEN" ) for i in "${required_vars[@]}" @@ -65,6 +65,8 @@ create_lv() log_info "start create lv:$LV_NAME" lvcreate -y -L 10G -n ${LV_NAME} ${VG_NAME} || die "create logical volume failed." + + mkfs.ext4 /dev/mapper/${VG_NAME}-${LV_NAME} } prepare_lv() @@ -111,7 +113,7 @@ generate_ks() logvol / --vgname=${VG_NAME} --name=${LV_NAME} --useexisting # Root password - rootpw --iscrypted ${ROOTFS_ROOT_PASSWD} + rootpw --iscrypted ${secrets_ROOTFS_ROOT_PASSWD} # Reboot after installation reboot --eject @@ -180,7 +182,7 @@ prepare_submit() SCHED_PORT: ${SCHED_PORT} my_name: ${my_name} my_email: ${my_email} - my_token: ${MY_TOKEN} + my_token: ${secrets_MY_TOKEN} lab: z9 EOF } @@ -195,7 +197,7 @@ prepare_install_iso_yaml() custom_ipxe="#!ipxe\\\\n" custom_ipxe="${custom_ipxe}\\\\ndhcp" custom_ipxe="${custom_ipxe}\\\\ninitrd ${tftp_root}/initrd.img" - custom_ipxe="${custom_ipxe}\\\\nkernel ${tftp_root}/vmlinux ro inst.geoloc=0 console=ttyS0 smmu.bypassdev=0x1000:0x17 smmu.bypassde1000:0x15 viff video=VGA-1:640x480-32@60me initrd=initrd.img ip=dhcp inst.ks=http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/os/kickstart/ks/${iso_os}_${iso_arch}_${iso_version}.cfg" + custom_ipxe="${custom_ipxe}\\\\nkernel ${tftp_root}/vmlinuz ro inst.geoloc=0 console=ttyS0 smmu.bypassdev=0x1000:0x17 smmu.bypassde1000:0x15 viff video=VGA-1:640x480-32@60me initrd=initrd.img ip=dhcp inst.ks=http://${HTTP_SERVER_HOST}:${HTTP_SERVER_PORT}/os/install/ks/${iso_os}_${iso_arch}_${iso_version}.ks" custom_ipxe="${custom_ipxe}\\\\nboot\\\\n" log_info "custom_ipxe: $custom_ipxe" @@ -244,13 +246,13 @@ submit_install_iso_sync_rootfs_yaml() ${LKP_SRC}/sbin/submit \ ${sync_rootfs_yaml} \ suite=install-iso-sync-rootfs-${ROOTFS_SUFFIX} \ - iso_os=${iso_os} \ - iso_arch=${iso_arch} \ - iso_version=${iso_version} \ - NFS_SERVER_HOST=${NFS_SERVER_HOST} \ - NFS_ROOT_DIR=${NFS_ROOT_DIR} \ - ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ - VG_NAME=${VG_NAME} \ + install-iso-sync-rootfs.iso_os=${iso_os} \ + install-iso-sync-rootfs.iso_arch=${iso_arch} \ + install-iso-sync-rootfs.iso_version=${iso_version} \ + install-iso-sync-rootfs.NFS_SERVER_HOST=${NFS_SERVER_HOST} \ + install-iso-sync-rootfs.NFS_ROOT_DIR=${NFS_ROOT_DIR} \ + install-iso-sync-rootfs.ROOTFS_SUFFIX=${ROOTFS_SUFFIX} \ + install-iso-sync-rootfs.VG_NAME=${VG_NAME} \ testbox=$testbox || die "submit ${sync_rootfs_yaml} failed" } -- 2.23.0

1 0

[PATCH compass-ci] lib/rpm-repo.rb: do not move to pub directory until test completed
by Li Ping 19 Apr '21

19 Apr '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- lib/rpm-repo.rb | 38 +++++++++++++++++++++++++------------- 1 file changed, 25 insertions(+), 13 deletions(-) diff --git a/lib/rpm-repo.rb b/lib/rpm-repo.rb index 658a5270..f2344dcc 100755 --- a/lib/rpm-repo.rb +++ b/lib/rpm-repo.rb @@ -31,29 +31,41 @@ MQ_PORT = ENV['MQ_PORT'] || 5672 class HandleRepo def initialize @mq = MQClient.new(MQ_HOST, MQ_PORT) - @update = [] end def handle_new_rpm queue = @mq.queue("update_repo") queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| - rpm_info = JSON.parse(msg) - rpm_info["upload_rpms"].each do |rpm| - rpm_path = File.dirname(rpm).sub("upload", "testing") - FileUtils.mkdir_p(rpm_path) unless File.directory?(rpm_path) + begin + rpm_info = JSON.parse(msg) + check_upload_rpms(rpm_info) + rpm_info["upload_rpms"].each do |rpm| + rpm_path = File.dirname(rpm).sub("upload", "testing") + FileUtils.mkdir_p(rpm_path) unless File.directory?(rpm_path) - dest = File.join(rpm_path.to_s, File.basename(rpm)) - @update << dest - FileUtils.mv(rpm, dest) - system("createrepo --update $(dirname #{rpm_path})") + dest = File.join(rpm_path.to_s, File.basename(rpm)) + FileUtils.mv(rpm, dest) + system("createrepo --update $(dirname #{rpm_path})") + end + @mq.ack(info) + rescue StandardError => e + puts "error message: ", e.message + @mq.ack(info) end - update_pub_dir - @mq.ack(info) end end - def update_pub_dir - @update.each do |rpm| + def check_upload_rpms(data) + raise JSON.dump({ 'errcode' => '200', 'errmsg' => 'no upload_rpms params' }) unless data.key?('upload_rpms') + raise JSON.dump({ 'errcode' => '200', 'errmsg' => "upload_rpms params type error" }) if data["upload_rpms"].class != Array + data["upload_rpms"].each do |rpm| + raise JSON.dump({ "errcode" => "200", "errmsg" => "#{rpm} not exist" }) unless File.exist?(rpm) + raise JSON.dump({ "errcode" => "200", "errmsg" => "the upload directory is incorrect" }) unless File.dirname(rpm).start_with?("/srv/rpm/upload/") + end + end + + def update_pub_dir(update) + update.each do |rpm| pub_path = File.dirname(rpm).sub("testing", "pub") FileUtils.mkdir_p(pub_path) unless File.directory?(pub_path) -- 2.23.0

1 0

[PATCH compass-ci] container/webhook: change the port which listen to
by Li Yuanchao 19 Apr '21

19 Apr '21

Change from 11301 to 20005 as planned Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/webhook/root/webhook.js | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/webhook/root/webhook.js b/container/webhook/root/webhook.js index 4639e14..6ca836e 100755 --- a/container/webhook/root/webhook.js +++ b/container/webhook/root/webhook.js @@ -17,4 +17,4 @@ http.createServer(function(req, res){ res.statusCode = 404 res.end('no such location') }) -}).listen(11301) +}).listen(20005) -- 2.23.0

1 0

[PATCH lkp-tests] pkg/mysql/PKGBUILD: fix md5sums
by Lin Jiaxin 19 Apr '21

19 Apr '21

reference: commit bc02d3fe0c13cf85d561a26fdef8dd04c8ede4aa Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- pkg/mysql/PKGBUILD | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/pkg/mysql/PKGBUILD b/pkg/mysql/PKGBUILD index 304f6a4a6..c13f07d67 100644 --- a/pkg/mysql/PKGBUILD +++ b/pkg/mysql/PKGBUILD @@ -9,7 +9,7 @@ source=("https://cdn.mysql.com/archives/mysql-8.0/mysql-boost-$pkgver.tar.gz" "https://mirrors.huaweicloud.com/kunpeng/archive/openEuler/patch/database/my…" "https://github.com/thkukuk/rpcsvc-proto/releases/download/v1.4/rpcsvc-proto…" "my.cnf") -md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' '1390631a1ea1832f9c27d07fcc1e536e') +md5sums=('7472a25d91973cbba5d8a8f176a3080b' '6e853850efbd6f887ac3e13b1bf99cc4' 'f13eb11de53616017f7428cc06d4e242' 'df1022ccc5d0aff30b8e505735b4a98b' 'd90ba2c701507eeac43b06fcfa49c2c1') prepare() { -- 2.23.0

1 0

[PATCH compass-ci] es_query.rb: fix _scroll_id change
by Cui Yili 19 Apr '21

19 Apr '21

[why] _scroll_id will change every time query, so you need to save the last _scroll_id. Signed-off-by: Cui Yili <2268260388(a)qq.com> --- lib/es_query.rb | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/lib/es_query.rb b/lib/es_query.rb index 22d68a9..6bd39bf 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -47,11 +47,12 @@ class ESQuery }, size: size } result = @client.search index: @index, scroll: '10m', body: query - @scroll_id = result['_scroll_id'] - return result else - @client.scroll scroll: '10m', scroll_id: @scroll_id + result = @client.scroll scroll: '10m', scroll_id: @scroll_id end + + @scroll_id = result['_scroll_id'] unless result.empty? + return result end def query_by_id(id) -- 2.23.0

2 1

[PATCH compass-ci] sbin/auto_submit: add author_email and committer_email
by Li Yuanchao 19 Apr '21

19 Apr '21

Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/auto_submit b/sbin/auto_submit index faebb2f..4adce89 100755 --- a/sbin/auto_submit +++ b/sbin/auto_submit @@ -55,6 +55,9 @@ class AutoSubmit real_argvs.push("upstream_dir=#{belong}") tag = %x(git -C #{mirror_dir} tag --points-at #{commit_id}) real_argvs.push("upstream_tag=#{tag}") unless tag.empty? + author_email = %x(git -C #{mirror_dir} log --format=%ae -1) + committer_email = %x(git -C #{mirror_dir} log --format=%ce -1) + real_argvs.push("author_email=#{author_email} committer_email=#{committer_email}") system(real_argvs.join(' ')) end -- 2.23.0

1 0

[PATCH compass-ci 2/2] scheduler: allow some variables to be used in the job
by Yu Chuan 19 Apr '21

19 Apr '21

[Why] allow two variables: - OS_HTTP_HOST - OS_HTTP_PORT Signed-off-by: Yu Chuan <13186087857(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 76686a5f5823..969889edc16b 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -151,6 +151,8 @@ SEND_MAIL_HOST: SEND_MAIL_PORT: INITRD_HTTP_HOST: INITRD_HTTP_PORT: +OS_HTTP_HOST: +OS_HTTP_PORT: ASSIST_RESULT_HOST: ASSIST_RESULT_PORT: ASSISTANT_HOST: -- 2.23.0

1 0

[PATCH compass-ci 1/2] scheduler: allow all jobs with suite=install-iso* to use custom_ipxe
by Yu Chuan 19 Apr '21

19 Apr '21

[Why] Job with suite=install-iso-firstboot needs use custom_ipxe feature. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- src/scheduler/find_job_boot.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 41fa89f0056b..1ccd9e6de644 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -287,7 +287,7 @@ class Sched end private def get_boot_ipxe(job : Job) - return job["custom_ipxe"] if job["suite"] == "install-iso" && job.has_key?("custom_ipxe") + return job["custom_ipxe"] if job["suite"].starts_with?("install-iso") && job.has_key?("custom_ipxe") response = "#!ipxe\n\n" -- 2.23.0

1 0

[PATCH compass-ci] add permission 775 lkp:lkp
by Liu Yinsi 17 Apr '21

17 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index db33a93..3c229c5 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -13,6 +13,9 @@ chmod_dir=( /srv/rabbitmq /c/git-repos /srv/tmp/public + /srv/initrd + /srv/initrd/pkg + /srv/initrd/deps ) chmod 775 "${chmod_dir[@]}" @@ -20,6 +23,8 @@ chmod 775 "${chmod_dir[@]}" chown_dir=( /srv/result /srv/initrd + /srv/initrd/pkg + /srv/initrd/deps /srv/es /srv/cache/netdata_cache /srv/cache/netdata_lib -- 2.23.0

1 0

[PATCH compass-ci] etcd/build: remove redundant spaces
by Wu Zhende 17 Apr '21

17 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/etcd/build | 2 -- 1 file changed, 2 deletions(-) diff --git a/container/etcd/build b/container/etcd/build index 6ae25dd..37c2326 100755 --- a/container/etcd/build +++ b/container/etcd/build @@ -8,5 +8,3 @@ version=v3.4.15 docker_skip_rebuild "etcd-${version}:alpine" docker build -t etcd-${version}:alpine --build-arg VERSION=${version} . - - -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: add subqueue
by Wu Zhende 17 Apr '21

17 Apr '21

Use subqueue to search Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 9926fd8..67b1ed1 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -100,6 +100,9 @@ else "queue": { "type": "keyword" }, + "subqueue": { + "type": "keyword" + }, "testbox": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] scheduler/scheduler.cr: add warning logs
by Wu Zhende 17 Apr '21

17 Apr '21

Because set "logging false", so need to print all the logs ourselves. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/scheduler.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index ae30937..19a3092 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -40,11 +40,15 @@ module Scheduler env.response.headers["Connection"] = "close" env.create_log env.create_sched + rescue e + env.log.warn(e.inspect_with_backtrace) end after_all do |env| env.sched.etcd_close env.log.info(%({"from": "#{env.request.remote_address}", "message": "access_record"})) + rescue e + env.log.warn(e.inspect_with_backtrace) end # echo alive -- 2.23.0

1 0

[PATCH compass-ci] lib/dump_stat.rb: add message key in cols verifation
by Lu Weitao 17 Apr '21

17 Apr '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 2 -- 1 file changed, 2 deletions(-) diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb index 04a7632..a497d46 100644 --- a/lib/dump_stat.rb +++ b/lib/dump_stat.rb @@ -121,8 +121,6 @@ module DumpStat @max_cols_stat = '' zero_stats = [] @result.each do |key, val| - next if key.end_with?('.message', '.log') - if @max_cols < val.size @max_cols = val.size @max_cols_stat = key -- 2.23.0

1 0

[PATCH compass-ci] add work dir /srv/tmp/public
by Liu Yinsi 17 Apr '21

17 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/tiny | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index 0eb7d39..c7636b0 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,6 +26,7 @@ dirs=( /srv/cci/Maildir/.compass-ci/cur /srv/cci/Maildir/.compass-ci/sent /srv/rabbitmq + /srv/tmp/public ) mkdir -p "${dirs[@]}" -- 2.23.0

1 0

[PATCH compass-ci] container/assist-result: refactor get_job_stats to get_job_content
by Cao Xueliang 17 Apr '21

17 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 6 +++--- .../views/{get_job_stats.rb => get_job_content.rb} | 5 ++--- lib/assist_result_client.rb | 8 ++++++++ src/delimiter/utils.rb | 7 +++++-- 4 files changed, 18 insertions(+), 8 deletions(-) rename container/assist-result/views/{get_job_stats.rb => get_job_content.rb} (67%) diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 02956aa..287db25 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,7 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' -require_relative './views/get_job_stats' +require_relative './views/get_job_content' require_relative './views/check_job_credible' configure do @@ -36,9 +36,9 @@ post '/check_job_credible' do return [200, {'credible' => result}.to_json] end -get '/get_job_stats/:job_id' do +get '/get_job_content/:job_id' do begin - result = get_job_stats(params[:job_id]) + result = get_job_content(params[:job_id]) rescue StandardError => e return [400, e.backtrace.inspect] end diff --git a/container/assist-result/views/get_job_stats.rb b/container/assist-result/views/get_job_content.rb similarity index 67% rename from container/assist-result/views/get_job_stats.rb rename to container/assist-result/views/get_job_content.rb index d294df3..643beb5 100644 --- a/container/assist-result/views/get_job_stats.rb +++ b/container/assist-result/views/get_job_content.rb @@ -4,7 +4,6 @@ require_relative "#{ENV['CCI_SRC']}/lib/es_query" -def get_job_stats(job_id) - content = ESQuery.new.query_by_id(job_id) - return content['stats'] +def get_job_content(job_id) + return ESQuery.new.query_by_id(job_id) end diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb index fcf5d9d..9c42b97 100644 --- a/lib/assist_result_client.rb +++ b/lib/assist_result_client.rb @@ -34,4 +34,12 @@ class AssistResult return JSON.parse(response.body) end + + def get_job_content(job_id) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_job_content/#{job_id}") + response = resource.get() + return nil unless response.code == 200 + + return response.body + end end diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index dbc42ef..a39d9de 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -108,9 +108,12 @@ module Utils # this is a temporary solution, the extract container will be improved in future. def query_stats(job_id, times) (1..times).each do |i| - new_job = ESQuery.new.query_by_id(job_id) + job_content = AssistResult.new.get_job_content(job_id) puts "query stats times: #{i}" - return new_job['stats'] if new_job['stats'] + if job_content + job_content = JSON.parse(job_content) + return job_content['stats'] if job_content.key?('stats') + end sleep 60 end -- 2.23.0

1 0

[PATCH v2 lkp-tests] script to deploy ceph for multi server tests
by Luan Shengde 16 Apr '21

16 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/ceph-init | 155 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100755 tests/ceph-init diff --git a/tests/ceph-init b/tests/ceph-init new file mode 100755 index 000000000..59398d1bd --- /dev/null +++ b/tests/ceph-init @@ -0,0 +1,155 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# - cephuser_password +# - cephuser + +: "${cephuser:=cephuser}" +: "${cephuser_password:=cephuser_password}" + +### ===== on each node ===== +# stop firewalld and disable selinx +systemctl stop firewalld +systemctl disable firewalld + +sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux + +until [[ -n $node_roles ]]; do + sleep 1 +done + +hosts=(node1 node2 node3) + +[[ $node_roles == node1 ]] && { + hostnamectl set-hostname node1 +} + +[[ $node_roles == node2 ]] && { + hostnamectl set-hostname node2 +} + +[[ $node_roles == node3 ]] && { + hostnamectl set-hostname node3 +} + +yum update && { + yum install ntp wget gcc make ceph lvm2 python3 -y +} + +# add user and set sudo permissions +useradd -d /home/$cephuser -m $cephuser && { + echo $cephuser:$cephuser_password | chpasswd +} + +cat >> /etc/sudoers <<-EOF +$cephuser ALL=(ALL) NOPASSWD:ALL +EOF + +# check and install pip +[[ $(command -v pip) ]] || { + until wget https://bootstrap.pypa.io/get-pip.py; do + sleep 2 + done + python get-pip.py +} + +pip install prettytable + +### ===== on ceph-deploy node ===== +# get node IPs is first +[[ $node_roles == node1 ]] || exit 0 + +until [[ -n $node1 ]]; do + sleep 1 +done + +until [[ -n $node2 ]]; do + sleep 1 +done + +until [[ -n $node3 ]]; do + sleep 1 +done + +# config hosts and set default ssh users +cat >> /etc/hosts <<-EOF +$node1 node1 node1 +$node2 node2 node2 +$node3 node3 node3 +EOF + +for host in "${hosts[@]}"; do + cat >> "$HOME/.ssh/config" <<-EOF + Host $host + Hostname $host + User $cephuser + EOF +done + +chmod 0600 "$HOME/.ssh/config" + +network=$(ip route | grep default | awk '{print $3}' | awk -F'.' '{print $1"."$2}') +net_mask=$(ip route | grep "^$network" | awk '{print $1}') + +# do the deploy until all the servers are up +# for host in "${host[@]}"; do +# until ping -c 1 -W 1 172.168.131.100> /dev/null; do +# sleep 2 +# done +# done + +# set server trust +[[ $(command -v sshpass) ]] || { + until wget http://sourceforge.net/projects/sshpass/files/latest/download -O sshpass.tar.gz; do + sleep 2 + done +} + +tar xzvf sshpass.tar.gz && { + cd sshpass-1.08 || exit 1 + ./configure + make + make install +} + +rm -rf "$HOME/.ssh/id_rsa" "$HOME/.ssh/id_rsa.pub" && { + ssh-keygen -f "$HOME/.ssh/id_rsa" -t rsa -N '' +} + +sed -i '/StrictHostKeyChecking/c StrictHostKeyChecking no' /etc/ssh/ssh_config +for host in "${hosts[@]}"; do +{ + sshpass -p $cephuser_password ssh-copy-id "$cephuser"@"$host" &> /dev/null +}& +done +wait + +# install +pip install ceph-deploy + +pip_dir=$(pip show ceph-deploy | awk '/^Location/ {print $NF}') +ceph_init_file="${pip_dir}/ceph_deploy/hosts/__init__.py" +[[ -f $ceph_init_file ]]&& { + sed -i "/distributions = {/a 'openeuler': fedora," "$ceph_init_file" +} + +# deploy ceph cluster +disk_name=$(sed -e '1d' -e '/^$/d' /proc/partitions | grep " .da" | awk '{print $NF}') +disk_device="/dev/${disk_name}" + +[[ -d "$HOME/ceph-deploy" ]] || { + mkdir -p "$HOME/ceph-deploy" +} + +cd "$HOME/ceph-deploy" && { + ceph-deploy new "${hosts[@]}" + echo "public_network = ${net_mask}" >> "$HOME/ceph-deploy/ceph.conf" + ceph-deploy mon create-initial + ceph-deploy --overwrite-conf admin "${hosts[@]}" + ceph-deploy mgr create "${hosts[@]}" + for host in "${hosts[@]}"; do + ceph-deploy osd create --data "$disk_device" "$host" + done + wait + ceph osd pool create ceph-osd-pool 128 +} -- 2.23.0

2 2

[PATCH v2 compass-ci] delimiter: consume bisect process job by the bad job tbox_group
by Cao Xueliang 16 Apr '21

16 Apr '21

Sometimes we set a queue in a job, then we delete the testbox who can consume the queue, if we submit the job to the queue, no testbox can consume it, so we consume the bisect process job by the bad job tbox_group. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index dbc42ef..1aac6b3 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -137,9 +137,10 @@ module Utils job['my_email'] = ENV['my_email'] job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id - job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" + job['testbox'] = job['tbox_group'] job.delete('id') + job.delete('queue') return job end -- 2.23.0

1 0

[PATCH v1 compass-ci 1/2] delimiter: consume job by the bad job queue
by Cao Xueliang 16 Apr '21

16 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 1 - 1 file changed, 1 deletion(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index dbc42ef..a0d4196 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -137,7 +137,6 @@ module Utils job['my_email'] = ENV['my_email'] job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id - job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" job.delete('id') -- 2.23.0

1 1

[PATCH v1 compass-ci 2/2] container/etcd: fix Error reading socket: Connection reset by peer
by Cao Xueliang 16 Apr '21

16 Apr '21

We should pass the start script params to /bin/etcd program. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/etcd/entry.sh | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/container/etcd/entry.sh b/container/etcd/entry.sh index 4bfea1e..16c8d00 100755 --- a/container/etcd/entry.sh +++ b/container/etcd/entry.sh @@ -3,6 +3,6 @@ # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. crond -/bin/etcd -exec "$@" +/bin/etcd "$@" +exec -- 2.23.0

1 0

[PATCH compass-ci 4/4] sparrow: expose ASSISTANT_PORT/HOST in scheduler
by Cao Xueliang 16 Apr '21

16 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- sparrow/3-code/dev-env | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index bb576c7..76686a5 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -28,6 +28,8 @@ server_name=$(hostname | cut -f1 -d.) : ${INITRD_HTTP_PORT:=8800} : ${assist_result_host:=$server_ip} : ${ASSIST_RESULT_PORT:=8102} +: ${assistant_host:=$server_ip} +: ${ASSISTANT_PORT:=8101} : ${result_webdav_host:=$server_ip} : ${RESULT_WEBDAV_PORT:=3080} : ${MASTER_FLUENTD_HOST:=$server_ip} @@ -107,6 +109,8 @@ INITRD_HTTP_HOST: $initrd_http_host INITRD_HTTP_PORT: $INITRD_HTTP_PORT ASSIST_RESULT_HOST: $assist_result_host ASSIST_RESULT_PORT: $ASSIST_RESULT_PORT +ASSISTANT_HOST: $assistant_host +ASSISTANT_PORT: $ASSISTANT_PORT RESULT_WEBDAV_HOST: $result_webdav_host RESULT_WEBDAV_PORT: $RESULT_WEBDAV_PORT MASTER_FLUENTD_HOST: $MASTER_FLUENTD_HOST @@ -149,6 +153,8 @@ INITRD_HTTP_HOST: INITRD_HTTP_PORT: ASSIST_RESULT_HOST: ASSIST_RESULT_PORT: +ASSISTANT_HOST: +ASSISTANT_PORT: RESULT_WEBDAV_HOST: RESULT_WEBDAV_PORT: DOCKER_REGISTRY_HOST: -- 2.23.0

1 0

[PATCH compass-ci 3/4] lib: add assistant_client and use it in mail_bisect_result
by Cao Xueliang 16 Apr '21

16 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- lib/assistant_client.rb | 24 ++++++++++++++++++++++++ lib/constants.rb | 3 +++ lib/mail_bisect_result.rb | 4 ++-- 3 files changed, 29 insertions(+), 2 deletions(-) create mode 100644 lib/assistant_client.rb diff --git a/lib/assistant_client.rb b/lib/assistant_client.rb new file mode 100644 index 0000000..c951f2b --- /dev/null +++ b/lib/assistant_client.rb @@ -0,0 +1,24 @@ +#!/usr/bin/env ruby +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# frozen_string_literal: true + +require 'json' +require 'base64' +require 'rest-client' +require_relative 'constants' + +class AssistantClient + def initialize(host = ASSISTANT_HOST, port = ASSISTANT_PORT) + @host = host + @port = port + end + + def get_mail_list(type) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_mail_list/#{type}") + response = resource.get() + return nil unless response.code == 200 + + return JSON.parse(response.body) + end +end diff --git a/lib/constants.rb b/lib/constants.rb index 03b1e40..11927f9 100644 --- a/lib/constants.rb +++ b/lib/constants.rb @@ -20,6 +20,9 @@ SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' +ASSISTANT_HOST ||= config['ASSISTANT_HOST'] || ENV['ASSISTANT_HOST'] || '172.17.0.1' +ASSISTANT_PORT ||= config['ASSISTANT_PORT'] || ENV['ASSISTANT_PORT'] || 8101 + ASSIST_RESULT_HOST ||= config['ASSIST_RESULT_HOST'] || ENV['ASSIST_RESULT_HOST'] || '172.17.0.1' ASSIST_RESULT_PORT ||= config['ASSIST_RESULT_PORT'] || ENV['ASSIST_RESULT_PORT'] || 8102 diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb index 782c702..5a71225 100644 --- a/lib/mail_bisect_result.rb +++ b/lib/mail_bisect_result.rb @@ -6,7 +6,7 @@ require_relative 'git' require_relative 'es_query' require_relative 'constants' require_relative 'mail_client' -require_relative 'parse_mail_list' +require_relative 'assistant_client' require_relative 'assign_account_client' # compose and send email for bisect result @@ -25,7 +25,7 @@ class MailBisectResult end def parse_mail_info - mail_hash = parse_mail_list('delimiter') + mail_hash = AssistantClient.new.get_mail_list('delimiter') @to = mail_hash['to'] if mail_hash.key?('to') @bcc = mail_hash['bcc'] if mail_hash.key?('bcc') raise 'Need to add bcc email for bisect report.' unless @bcc -- 2.23.0

1 0