- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 1/2] scheduler: allow all jobs with suite=install-iso* to use custom_ipxe
by Yu Chuan 19 Apr '21

19 Apr '21

[Why] Job with suite=install-iso-firstboot needs use custom_ipxe feature. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- src/scheduler/find_job_boot.cr | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index 41fa89f0056b..1ccd9e6de644 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -287,7 +287,7 @@ class Sched end private def get_boot_ipxe(job : Job) - return job["custom_ipxe"] if job["suite"] == "install-iso" && job.has_key?("custom_ipxe") + return job["custom_ipxe"] if job["suite"].starts_with?("install-iso") && job.has_key?("custom_ipxe") response = "#!ipxe\n\n" -- 2.23.0

1 0

[PATCH compass-ci] add permission 775 lkp:lkp
by Liu Yinsi 17 Apr '21

17 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/permission | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/sparrow/1-storage/permission b/sparrow/1-storage/permission index db33a93..3c229c5 100755 --- a/sparrow/1-storage/permission +++ b/sparrow/1-storage/permission @@ -13,6 +13,9 @@ chmod_dir=( /srv/rabbitmq /c/git-repos /srv/tmp/public + /srv/initrd + /srv/initrd/pkg + /srv/initrd/deps ) chmod 775 "${chmod_dir[@]}" @@ -20,6 +23,8 @@ chmod 775 "${chmod_dir[@]}" chown_dir=( /srv/result /srv/initrd + /srv/initrd/pkg + /srv/initrd/deps /srv/es /srv/cache/netdata_cache /srv/cache/netdata_lib -- 2.23.0

1 0

[PATCH compass-ci] etcd/build: remove redundant spaces
by Wu Zhende 17 Apr '21

17 Apr '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- container/etcd/build | 2 -- 1 file changed, 2 deletions(-) diff --git a/container/etcd/build b/container/etcd/build index 6ae25dd..37c2326 100755 --- a/container/etcd/build +++ b/container/etcd/build @@ -8,5 +8,3 @@ version=v3.4.15 docker_skip_rebuild "etcd-${version}:alpine" docker build -t etcd-${version}:alpine --build-arg VERSION=${version} . - - -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-jobs-mapping.sh: add subqueue
by Wu Zhende 17 Apr '21

17 Apr '21

Use subqueue to search Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- sbin/es-jobs-mapping.sh | 3 +++ 1 file changed, 3 insertions(+) diff --git a/sbin/es-jobs-mapping.sh b/sbin/es-jobs-mapping.sh index 9926fd8..67b1ed1 100755 --- a/sbin/es-jobs-mapping.sh +++ b/sbin/es-jobs-mapping.sh @@ -100,6 +100,9 @@ else "queue": { "type": "keyword" }, + "subqueue": { + "type": "keyword" + }, "testbox": { "type": "keyword" }, -- 2.23.0

1 0

[PATCH compass-ci] scheduler/scheduler.cr: add warning logs
by Wu Zhende 17 Apr '21

17 Apr '21

Because set "logging false", so need to print all the logs ourselves. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/scheduler.cr | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/scheduler/scheduler.cr b/src/scheduler/scheduler.cr index ae30937..19a3092 100644 --- a/src/scheduler/scheduler.cr +++ b/src/scheduler/scheduler.cr @@ -40,11 +40,15 @@ module Scheduler env.response.headers["Connection"] = "close" env.create_log env.create_sched + rescue e + env.log.warn(e.inspect_with_backtrace) end after_all do |env| env.sched.etcd_close env.log.info(%({"from": "#{env.request.remote_address}", "message": "access_record"})) + rescue e + env.log.warn(e.inspect_with_backtrace) end # echo alive -- 2.23.0

1 0

[PATCH compass-ci] lib/dump_stat.rb: add message key in cols verifation
by Lu Weitao 17 Apr '21

17 Apr '21

Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- lib/dump_stat.rb | 2 -- 1 file changed, 2 deletions(-) diff --git a/lib/dump_stat.rb b/lib/dump_stat.rb index 04a7632..a497d46 100644 --- a/lib/dump_stat.rb +++ b/lib/dump_stat.rb @@ -121,8 +121,6 @@ module DumpStat @max_cols_stat = '' zero_stats = [] @result.each do |key, val| - next if key.end_with?('.message', '.log') - if @max_cols < val.size @max_cols = val.size @max_cols_stat = key -- 2.23.0

1 0

[PATCH compass-ci] add work dir /srv/tmp/public
by Liu Yinsi 17 Apr '21

17 Apr '21

Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/1-storage/tiny | 1 + 1 file changed, 1 insertion(+) diff --git a/sparrow/1-storage/tiny b/sparrow/1-storage/tiny index 0eb7d39..c7636b0 100755 --- a/sparrow/1-storage/tiny +++ b/sparrow/1-storage/tiny @@ -26,6 +26,7 @@ dirs=( /srv/cci/Maildir/.compass-ci/cur /srv/cci/Maildir/.compass-ci/sent /srv/rabbitmq + /srv/tmp/public ) mkdir -p "${dirs[@]}" -- 2.23.0

1 0

[PATCH compass-ci] container/assist-result: refactor get_job_stats to get_job_content
by Cao Xueliang 17 Apr '21

17 Apr '21

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- container/assist-result/routes.rb | 6 +++--- .../views/{get_job_stats.rb => get_job_content.rb} | 5 ++--- lib/assist_result_client.rb | 8 ++++++++ src/delimiter/utils.rb | 7 +++++-- 4 files changed, 18 insertions(+), 8 deletions(-) rename container/assist-result/views/{get_job_stats.rb => get_job_content.rb} (67%) diff --git a/container/assist-result/routes.rb b/container/assist-result/routes.rb index 02956aa..287db25 100755 --- a/container/assist-result/routes.rb +++ b/container/assist-result/routes.rb @@ -7,7 +7,7 @@ require 'json' require 'sinatra' require_relative './views/get_job_yaml' -require_relative './views/get_job_stats' +require_relative './views/get_job_content' require_relative './views/check_job_credible' configure do @@ -36,9 +36,9 @@ post '/check_job_credible' do return [200, {'credible' => result}.to_json] end -get '/get_job_stats/:job_id' do +get '/get_job_content/:job_id' do begin - result = get_job_stats(params[:job_id]) + result = get_job_content(params[:job_id]) rescue StandardError => e return [400, e.backtrace.inspect] end diff --git a/container/assist-result/views/get_job_stats.rb b/container/assist-result/views/get_job_content.rb similarity index 67% rename from container/assist-result/views/get_job_stats.rb rename to container/assist-result/views/get_job_content.rb index d294df3..643beb5 100644 --- a/container/assist-result/views/get_job_stats.rb +++ b/container/assist-result/views/get_job_content.rb @@ -4,7 +4,6 @@ require_relative "#{ENV['CCI_SRC']}/lib/es_query" -def get_job_stats(job_id) - content = ESQuery.new.query_by_id(job_id) - return content['stats'] +def get_job_content(job_id) + return ESQuery.new.query_by_id(job_id) end diff --git a/lib/assist_result_client.rb b/lib/assist_result_client.rb index fcf5d9d..9c42b97 100644 --- a/lib/assist_result_client.rb +++ b/lib/assist_result_client.rb @@ -34,4 +34,12 @@ class AssistResult return JSON.parse(response.body) end + + def get_job_content(job_id) + resource = RestClient::Resource.new("http://#{@host}:#{@port}/get_job_content/#{job_id}") + response = resource.get() + return nil unless response.code == 200 + + return response.body + end end diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index dbc42ef..a39d9de 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -108,9 +108,12 @@ module Utils # this is a temporary solution, the extract container will be improved in future. def query_stats(job_id, times) (1..times).each do |i| - new_job = ESQuery.new.query_by_id(job_id) + job_content = AssistResult.new.get_job_content(job_id) puts "query stats times: #{i}" - return new_job['stats'] if new_job['stats'] + if job_content + job_content = JSON.parse(job_content) + return job_content['stats'] if job_content.key?('stats') + end sleep 60 end -- 2.23.0

1 0

[PATCH v2 lkp-tests] script to deploy ceph for multi server tests
by Luan Shengde 16 Apr '21

16 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/ceph-init | 155 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 155 insertions(+) create mode 100755 tests/ceph-init diff --git a/tests/ceph-init b/tests/ceph-init new file mode 100755 index 000000000..59398d1bd --- /dev/null +++ b/tests/ceph-init @@ -0,0 +1,155 @@ +#!/bin/bash +# SPDX-License-Identifier: MulanPSL-2.0+ +# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. +# - cephuser_password +# - cephuser + +: "${cephuser:=cephuser}" +: "${cephuser_password:=cephuser_password}" + +### ===== on each node ===== +# stop firewalld and disable selinx +systemctl stop firewalld +systemctl disable firewalld + +sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux + +until [[ -n $node_roles ]]; do + sleep 1 +done + +hosts=(node1 node2 node3) + +[[ $node_roles == node1 ]] && { + hostnamectl set-hostname node1 +} + +[[ $node_roles == node2 ]] && { + hostnamectl set-hostname node2 +} + +[[ $node_roles == node3 ]] && { + hostnamectl set-hostname node3 +} + +yum update && { + yum install ntp wget gcc make ceph lvm2 python3 -y +} + +# add user and set sudo permissions +useradd -d /home/$cephuser -m $cephuser && { + echo $cephuser:$cephuser_password | chpasswd +} + +cat >> /etc/sudoers <<-EOF +$cephuser ALL=(ALL) NOPASSWD:ALL +EOF + +# check and install pip +[[ $(command -v pip) ]] || { + until wget https://bootstrap.pypa.io/get-pip.py; do + sleep 2 + done + python get-pip.py +} + +pip install prettytable + +### ===== on ceph-deploy node ===== +# get node IPs is first +[[ $node_roles == node1 ]] || exit 0 + +until [[ -n $node1 ]]; do + sleep 1 +done + +until [[ -n $node2 ]]; do + sleep 1 +done + +until [[ -n $node3 ]]; do + sleep 1 +done + +# config hosts and set default ssh users +cat >> /etc/hosts <<-EOF +$node1 node1 node1 +$node2 node2 node2 +$node3 node3 node3 +EOF + +for host in "${hosts[@]}"; do + cat >> "$HOME/.ssh/config" <<-EOF + Host $host + Hostname $host + User $cephuser + EOF +done + +chmod 0600 "$HOME/.ssh/config" + +network=$(ip route | grep default | awk '{print $3}' | awk -F'.' '{print $1"."$2}') +net_mask=$(ip route | grep "^$network" | awk '{print $1}') + +# do the deploy until all the servers are up +# for host in "${host[@]}"; do +# until ping -c 1 -W 1 172.168.131.100> /dev/null; do +# sleep 2 +# done +# done + +# set server trust +[[ $(command -v sshpass) ]] || { + until wget http://sourceforge.net/projects/sshpass/files/latest/download -O sshpass.tar.gz; do + sleep 2 + done +} + +tar xzvf sshpass.tar.gz && { + cd sshpass-1.08 || exit 1 + ./configure + make + make install +} + +rm -rf "$HOME/.ssh/id_rsa" "$HOME/.ssh/id_rsa.pub" && { + ssh-keygen -f "$HOME/.ssh/id_rsa" -t rsa -N '' +} + +sed -i '/StrictHostKeyChecking/c StrictHostKeyChecking no' /etc/ssh/ssh_config +for host in "${hosts[@]}"; do +{ + sshpass -p $cephuser_password ssh-copy-id "$cephuser"@"$host" &> /dev/null +}& +done +wait + +# install +pip install ceph-deploy + +pip_dir=$(pip show ceph-deploy | awk '/^Location/ {print $NF}') +ceph_init_file="${pip_dir}/ceph_deploy/hosts/__init__.py" +[[ -f $ceph_init_file ]]&& { + sed -i "/distributions = {/a 'openeuler': fedora," "$ceph_init_file" +} + +# deploy ceph cluster +disk_name=$(sed -e '1d' -e '/^$/d' /proc/partitions | grep " .da" | awk '{print $NF}') +disk_device="/dev/${disk_name}" + +[[ -d "$HOME/ceph-deploy" ]] || { + mkdir -p "$HOME/ceph-deploy" +} + +cd "$HOME/ceph-deploy" && { + ceph-deploy new "${hosts[@]}" + echo "public_network = ${net_mask}" >> "$HOME/ceph-deploy/ceph.conf" + ceph-deploy mon create-initial + ceph-deploy --overwrite-conf admin "${hosts[@]}" + ceph-deploy mgr create "${hosts[@]}" + for host in "${hosts[@]}"; do + ceph-deploy osd create --data "$disk_device" "$host" + done + wait + ceph osd pool create ceph-osd-pool 128 +} -- 2.23.0

2 2

[PATCH v2 compass-ci] delimiter: consume bisect process job by the bad job tbox_group
by Cao Xueliang 16 Apr '21

16 Apr '21

Sometimes we set a queue in a job, then we delete the testbox who can consume the queue, if we submit the job to the queue, no testbox can consume it, so we consume the bisect process job by the bad job tbox_group. Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/delimiter/utils.rb | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/delimiter/utils.rb b/src/delimiter/utils.rb index dbc42ef..1aac6b3 100644 --- a/src/delimiter/utils.rb +++ b/src/delimiter/utils.rb @@ -137,9 +137,10 @@ module Utils job['my_email'] = ENV['my_email'] job['my_token'] = ENV['secrets_my_token'] job['bad_job_id'] = job_id - job['queue'] = "#{job['testbox'].split('.')[0]}-bisect" + job['testbox'] = job['tbox_group'] job.delete('id') + job.delete('queue') return job end -- 2.23.0

1 0