- Compass-ci - mailweb.openeuler.org

[PATCH lkp-tests] tests/rpmbuild-pkg: adjusting code for auto_submit
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- tests/rpmbuild-pkg | 23 +++++++++-------------- 1 file changed, 9 insertions(+), 14 deletions(-) diff --git a/tests/rpmbuild-pkg b/tests/rpmbuild-pkg index a8c8eb6b1..f49ba3a0b 100755 --- a/tests/rpmbuild-pkg +++ b/tests/rpmbuild-pkg @@ -1,11 +1,11 @@ #!/bin/bash -# - rpm_repo +# - upstream_repo . $LKP_SRC/lib/debug.sh . $LKP_SRC/lib/upload.sh -[ -n "$rpm_repo" ] || die "rpm_repo is empty" -package_name=${rpm_repo##*/} +[ -n "$upstream_repo" ] || die "upstream_repo is empty" +package_name=${upstream_repo##*/} sync_dest="/initrd/rpmbuild-pkg/${os}-${os_version}/${os_arch}/${package_name}" init_workspace() @@ -14,21 +14,16 @@ init_workspace() rpmdev-setuptree } -get_pkgfile() +download_upstream_repo() { - curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-show", "HEAD:'$1'"]}' -o "${2}" -} - -download_rpm_repo() -{ - local filelist=$(curl -sS -H 'Content-Type: Application/json' -XPOST "$GIT_SERVER"':8100/git_command' \ - -d '{"git_repo": "'${rpm_repo}'", "git_command": ["git-ls-files", "."]}') + git clone "git://$GIT_SERVER/openeuler/${upstream_repo}" || die "clone ${package_name} git repo failed: git://$GIT_SERVER/openeuler/${upstream_repo}" + cd $package_name || exit + filelist=$(git ls-files) for pkgfile in ${filelist[@]} do local dir="SOURCES" echo $pkgfile | egrep "\.spec$" && dir="SPECS" - get_pkgfile "$pkgfile" "${HOME}/rpmbuild/${dir}/$pkgfile" + cat "$pkgfile" > "${HOME}/rpmbuild/${dir}/$pkgfile" done } @@ -63,6 +58,6 @@ upload_rpm_pkg() } init_workspace -download_rpm_repo +download_upstream_repo build_rpm upload_rpm_pkg -- 2.23.0

2 2

[PATCH compass-ci] container/dracut-initrd: add basename into openeuler's 21xx+ kernel
by Xu Xijian 12 Mar '21

12 Mar '21

[why] It seems openeuler 21xx+ kernel not support cmd basename, our overlay need this cmd, so add basename into dracut module to enable initrd. [errlog] //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 50: basename: command not found //lib/dracut/hooks/pre-pivot/10-overlay-mount.sh: line 65: basename: command not found Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- container/dracut-initrd/bin/setup-dracut.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/dracut-initrd/bin/setup-dracut.sh b/container/dracut-initrd/bin/setup-dracut.sh index 45e895c..494588a 100755 --- a/container/dracut-initrd/bin/setup-dracut.sh +++ b/container/dracut-initrd/bin/setup-dracut.sh @@ -17,7 +17,7 @@ cp -a /usr/local/bin/cifs-lib.sh /usr/lib/dracut/modules.d/95cifs/ cat overlay-lkp.sh >> /usr/lib/dracut/modules.d/90overlay-root/overlay-mount.sh sed -i "/install() {/a\ inst /usr/bin/awk" /usr/lib/dracut/modules.d/40network/module-setup.sh -sed -i "/install() {/a\ inst /sbin/mke2fs" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh +sed -i "/install() {/a\ inst /sbin/mke2fs /usr/bin/basename" /usr/lib/dracut/modules.d/98dracut-systemd/module-setup.sh pre_mount_file="/usr/lib/dracut/modules.d/98dracut-systemd/dracut-pre-mount.sh" [ "$(sed -n '$p' $pre_mount_file)" = "exit 0" ] && sed -i '$d' "$pre_mount_file" cat set-local-sysroot.sh >> "$pre_mount_file" -- 2.23.0

2 2

[PATCH v2 compass-ci] src/lib/job.cr: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2177b22..9e18a4c 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -277,11 +277,8 @@ class Job # same usage for client mount_type = "nfs" if mount_type == "cifs" - if @hash["rpmbuild-pkg"]? - common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - else - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - end + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash.has_key?("rpmbuild-pkg") return common_dir end -- 2.23.0

1 0

[PATCH compass-ci] src/lib/job.cr: change rpmbuild-pkg upload_dir
by Li Ping 12 Mar '21

12 Mar '21

Signed-off-by: Li Ping <1477412247(a)qq.com> --- src/lib/job.cr | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 2177b22..1fbf646 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -277,11 +277,8 @@ class Job # same usage for client mount_type = "nfs" if mount_type == "cifs" - if @hash["rpmbuild-pkg"]? - common_dir = "#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - else - common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" - end + common_dir = "#{mount_type}/#{tmp_os}/#{tmp_os_arch}/#{tmp_os_version}" + common_dir = "#{tmp_os}-#{tmp_os_version}/#{tmp_os_arch}" if @hash["rpmbuild-pkg"]? return common_dir end -- 2.23.0

2 2

[PATCH v3 compass-ci 2/2] multi-docker: add reboot function
by Wu Zhende 12 Mar '21

12 Mar '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- providers/docker/docker.rb | 34 +++++++++++++++++++++++++++++++++- providers/docker/run.sh | 2 ++ 2 files changed, 35 insertions(+), 1 deletion(-) diff --git a/providers/docker/docker.rb b/providers/docker/docker.rb index f4e954e..9d6854f 100755 --- a/providers/docker/docker.rb +++ b/providers/docker/docker.rb @@ -7,6 +7,8 @@ require 'open-uri' require 'json' require 'set' require 'fileutils' + +require '../lib/mq_client' require_relative '../../container/defconfig' BASE_DIR = '/srv/dc' @@ -19,6 +21,8 @@ defaults = relevant_defaults(names) SCHED_HOST = defaults['SCHED_HOST'] || '172.17.0.1' SCHED_PORT = defaults['SCHED_PORT'] || 3000 LOG_DIR = '/srv/cci/serial/logs' +MQ_HOST = ENV['MQ_HOST'] || ENV['LKP_SERVER'] || 'localhost' +MQ_PORT = ENV['MQ_PORT'] || 5672 def get_url(hostname) "http://#{SCHED_HOST}:#{SCHED_PORT}/boot.container/hostname/#{hostname}" @@ -84,7 +88,8 @@ def start_container(hostname, load_path, hash) docker_image = hash['docker_image'] system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" system( - { 'hostname' => hostname, + { 'job_id' => hash['job_id'], + 'hostname' => hostname, 'docker_image' => docker_image, 'load_path' => load_path, 'log_dir' => "#{LOG_DIR}/#{hostname}" @@ -145,6 +150,29 @@ def loop_main(hostname, queues) end end +def loop_reboot_docker(hostname) + loop do + begin + reboot_docker(hostname) + rescue StandardError => e + puts e.backtrace + sleep 5 + end + end +end + +def reboot_docker(hostname) + mq = MQClient.new(MQ_HOST, MQ_PORT) + queue = mq.queue(hostname, {:durable => true}) + queue.subscribe({:block => true, :manual_ack => true}) do |info, _pro, msg| + puts msg + machine_info = JSON.parse(msg) + job_id = machine_info['job_id'] + system "docker rm -f #{job_id};echo $?" + mq.ack(info) + end +end + def save_pid(pids) FileUtils.cd("#{ENV['CCI_SRC']}/providers") f = File.new('dc.pid', 'a') @@ -153,6 +181,10 @@ def save_pid(pids) end def multi_docker(hostname, nr_container, queues) + Process.fork do + loop_reboot_docker(hostname) + end + pids = [] nr_container.to_i.times do |i| pid = Process.fork do diff --git a/providers/docker/run.sh b/providers/docker/run.sh index 2dfe36d..80950dd 100755 --- a/providers/docker/run.sh +++ b/providers/docker/run.sh @@ -4,6 +4,7 @@ . $LKP_SRC/lib/yaml.sh +: ${job_id:=$$} : ${docker_image:="centos:7"} : ${load_path:="${HOME}/jobs"} : ${hostname:="dc-1g-1"} @@ -23,6 +24,7 @@ busybox_path=$(command -v busybox) cmd=( docker run --rm + --name ${job_id} --hostname $host.compass-ci.net -m $memory --tmpfs /tmp:rw,exec,nosuid,nodev -- 2.23.0

1 0

[PATCH v3 compass-ci 1/2] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH v2 compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] extract-stats: wait_for refresh doc to ES DB
by Lu Weitao 12 Mar '21

12 Mar '21

[Why] We need update doc(job) to ES completed, then send delimiter task to task-queue Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- src/extract-stats/stats_worker.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/extract-stats/stats_worker.cr b/src/extract-stats/stats_worker.cr index dbb3fe2..44c327d 100644 --- a/src/extract-stats/stats_worker.cr +++ b/src/extract-stats/stats_worker.cr @@ -81,6 +81,7 @@ class StatsWorker { :index => "jobs", :type => "_doc", :id => job_id, + :refresh => "wait_for", :body => {:doc => update_content}, } ) -- 2.23.0

1 0

[PATCH compass-ci] scheduler/find_job_boot.cr: docker add job_id
by Wu Zhende 12 Mar '21

12 Mar '21

When the startup parameter is docker, the value of job_id is returned. Set job_id as docker name, so we can use "docker rm -f #{job_id}" to kill the docker. Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/scheduler/find_job_boot.cr | 1 + 1 file changed, 1 insertion(+) diff --git a/src/scheduler/find_job_boot.cr b/src/scheduler/find_job_boot.cr index ce1a2f8..a45a6d9 100644 --- a/src/scheduler/find_job_boot.cr +++ b/src/scheduler/find_job_boot.cr @@ -144,6 +144,7 @@ class Sched private def get_boot_container(job : Job) response = Hash(String, String).new + response["job_id"] = job.id.to_s response["docker_image"] = "#{job.docker_image}" response["lkp"] = "http://#{INITRD_HTTP_HOST}:#{INITRD_HTTP_PORT}" + JobHelper.service_path("#{SRV_INITRD}/lkp/#{job.lkp_initrd_user}/lkp-#{job.arch}.cgz") -- 2.23.0

1 0

[PATCH compass-ci] sparrow/3-code/dev-env: fix failed to submit job
by Liu Yinsi 12 Mar '21

12 Mar '21

[error] root@taishan200-2280-2s64p-256g--a123 /c/compass-ci/container# submit host-info.yaml submit_id=9bca6036-4602-44ab-92df-729058ac1818 Traceback (most recent call last): 5: from /c/lkp-tests/sbin/submit:136:in `<main>' 4: from /c/lkp-tests/sbin/submit:136:in `each' 3: from /c/lkp-tests/sbin/submit:150:in `block in <main>' 2: from /c/lkp-tests/lib/job.rb:744:in `each_jobs' 1: from /c/lkp-tests/lib/job.rb:736:in `each_job' /c/lkp-tests/sbin/submit:151:in `block (2 levels) in <main>': Please configure SCHED_HOST (RuntimeError) [why] for user locally deploy compass-ci, env $SCHED_HOST not exists in /etc/compass-ci/defaults/$server_name.yaml, cause failed to submit, so add it. Signed-off-by: Liu Yinsi <liuyinsi(a)163.com> --- sparrow/3-code/dev-env | 2 ++ 1 file changed, 2 insertions(+) diff --git a/sparrow/3-code/dev-env b/sparrow/3-code/dev-env index 403ccb1..fdf7894 100755 --- a/sparrow/3-code/dev-env +++ b/sparrow/3-code/dev-env @@ -36,6 +36,8 @@ server_name=$(hostname | cut -f1 -d.) mkdir -p /etc/compass-ci/defaults cat > /etc/compass-ci/defaults/$server_name.yaml <<EOF +SCHED_HOST: $sched_host +SCHED_PORT: $sched_port lab: nolab EOF -- 2.23.0

1 0