- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci] create-image: fix error path
by Wang Chenglong 02 Feb '21

02 Feb '21

Signed-off-by: Wang Chenglong <18509160991(a)163.com> --- container/osimage/openeuler/create-image | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/container/osimage/openeuler/create-image b/container/osimage/openeuler/create-image index 7f3d982..ed0bc14 100755 --- a/container/osimage/openeuler/create-image +++ b/container/osimage/openeuler/create-image @@ -37,7 +37,7 @@ setup_login() { pack_cgz() { echo "Packing package. Please wait." - find / ! -path "/tmp/${IMAGE_NAME}" | grep -vf /tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > /tmp/${IMAGE_NAME} + find ./ ! -path "./tmp/${IMAGE_NAME}" | grep -vf ./tmp/files-to-exclude | cpio -o -Hnewc | gzip -9 > ./tmp/${IMAGE_NAME} chmod 644 /tmp/${IMAGE_NAME} } -- 2.23.0

1 0

[PATCH lkp-tests] makepkg: avoid some unnecessary stderr
by Lin Jiaxin 02 Feb '21

02 Feb '21

unnecessary stder like this: ==> /tmp/stderr <== % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed ^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0^M10 0 129 0 0 137 0 --:--:-- --:--:-- --:--:-- 137 ^M 0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0^M100 255k 0 255k 0 0 110k 0 --:--:-- 0:00:02 --:--:-- 313k ==> WARNING: Skipping verification of source file PGP signatures. ==> /tmp/stderr <== fcgi-2.4.2.tar.gz ... Passed ==> /tmp/stderr <== Cloning into bare repository '//shadowsocks-libev-git'... ==> /tmp/stderr <== shadowsocks-libev-git ... Skipped shadowsocks-libev@.service ... Passed shadowsocks-libev-server@.service ... Passed shadowsocks-libev-redir@.service ... Passed shadowsocks-libev-tunnel@.service ... Passed Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- etc/makepkg.conf | 12 ++++++------ sbin/makepkg | 18 +++++++++--------- 2 files changed, 15 insertions(+), 15 deletions(-) diff --git a/etc/makepkg.conf b/etc/makepkg.conf index 5d73399d..b2beaedb 100644 --- a/etc/makepkg.conf +++ b/etc/makepkg.conf @@ -8,12 +8,12 @@ # #-- The download utilities that makepkg should use to acquire sources # Format: 'protocol::agent' -DLAGENTS=('file::/usr/bin/my_curl -gqC - -o %o %u' - 'ftp::/usr/bin/my_curl -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' - 'http::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'https::/usr/bin/my_curl -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' - 'rsync::/usr/bin/rsync --no-motd -z %u %o' - 'scp::/usr/bin/scp -C %u %o') +DLAGENTS=('file::/usr/bin/my_curl -sS -gqC - -o %o %u' + 'ftp::/usr/bin/my_curl -sS -gqfC - --ftp-pasv --retry 3 --retry-delay 3 -o %o %u' + 'http::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'https::/usr/bin/my_curl -sS -gqb "" -fLC - --retry 3 --retry-delay 3 -o %o %u' + 'rsync::/usr/bin/rsync -q --no-motd -z %u %o' + 'scp::/usr/bin/scp -q -C %u %o') # Other common tools: # /usr/bin/snarf diff --git a/sbin/makepkg b/sbin/makepkg index 498b4872..7a92ea71 100755 --- a/sbin/makepkg +++ b/sbin/makepkg @@ -566,7 +566,7 @@ download_git() { if [[ ! -d "$dir" ]] || dir_is_empty "$dir" ; then msg2 "$(gettext "Cloning %s %s repo...")" "${repo}" "git" - if ! git clone --mirror "$url" "$dir"; then + if ! git clone -q --mirror "$url" "$dir"; then error "$(gettext "Failure while downloading %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -580,7 +580,7 @@ download_git() { exit 1 fi msg2 "$(gettext "Updating %s %s repo...")" "${repo}" "git" - if ! git fetch --all -p; then + if ! git fetch -q --all -p; then # only warn on failure to allow offline builds warning "$(gettext "Failure while updating %s %s repo")" "${repo}" "git" fi @@ -609,13 +609,13 @@ extract_git() { if [[ -d "${dir##*/}" ]]; then updating=1 cd_safe "${dir##*/}" - if ! git fetch; then + if ! git fetch -q; then error "$(gettext "Failure while updating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 fi cd_safe "$srcdir" - elif ! git clone "$dir"; then + elif ! git clone -q "$dir"; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -645,7 +645,7 @@ extract_git() { fi if [[ $ref != "origin/HEAD" ]] || (( updating )) ; then - if ! git checkout --force --no-track -B makepkg $ref; then + if ! git checkout -q --force --no-track -B makepkg $ref; then error "$(gettext "Failure while creating working copy of %s %s repo")" "${repo}" "git" plain "$(gettext "Aborting...")" exit 1 @@ -1270,10 +1270,10 @@ verify_integrity_one() { local source_name=$1 integ=$2 expectedsum=$3 local file="$(get_filename "$source_name")" - printf ' %s ... ' "$file" >&2 + printf ' %s ... ' "$file" if [[ $expectedsum = 'SKIP' ]]; then - printf '%s\n' "$(gettext "Skipped")" >&2 + printf '%s\n' "$(gettext "Skipped")" return fi @@ -1285,7 +1285,7 @@ verify_integrity_one() { local realsum="$(openssl dgst -${integ} "$file")" realsum="${realsum##* }" if [[ ${expectedsum,,} = "$realsum" ]]; then - printf '%s\n' "$(gettext "Passed")" >&2 + printf '%s\n' "$(gettext "Passed")" else printf '%s\n' "$(gettext "FAILED")" >&2 return 1 @@ -1553,7 +1553,7 @@ check_source_integrity() { warning "$(gettext "Skipping verification of source file checksums.")" check_pgpsigs "$@" elif (( SKIPPGPCHECK )); then - warning "$(gettext "Skipping verification of source file PGP signatures.")" + # warning "$(gettext "Skipping verification of source file PGP signatures.")" check_checksums "$@" else check_checksums "$@" -- 2.23.0

2 1

[PATCH compass-ci] lib/git_mirror.rb: fix not use latest git remote url
by Lin Jiaxin 01 Feb '21

01 Feb '21

[why] When upstream-repos' url has changed https://git.archlinux.org/linux ==> https://mirrors.tuna.tsinghua.edu.cn/git/linux.git The bare git repo's remote url has not changed together. /srv/git/l/linux/linux.git% git ls-remote --get-url origin https://git.archlinux.org/linux Because the original logic only git fetch when git repo is existed. Now add check for git url. If git url has changed, rm the original git repo and clone again. Signed-off-by: Lin Jiaxin <ljx.joe(a)qq.com> --- lib/git_mirror.rb | 25 +++++++++++++++++++------ 1 file changed, 19 insertions(+), 6 deletions(-) diff --git a/lib/git_mirror.rb b/lib/git_mirror.rb index 13cb7c0..84a908b 100644 --- a/lib/git_mirror.rb +++ b/lib/git_mirror.rb @@ -63,16 +63,29 @@ class GitMirror return fetch_info.include? '->' end + def git_url_check(url, mirror_dir) + url = get_url(Array(url)[0]) + git_url = %x(git -C #{mirror_dir} ls-remote --get-url origin).chomp + + return true if url == git_url + + return false + end + + def git_repo_download(url, mirror_dir) + return git_clone(url, mirror_dir) unless File.directory?(mirror_dir) + + return git_fetch(mirror_dir) if git_url_check(url, mirror_dir) + + FileUtils.rm_r(mirror_dir) + return git_clone(url, mirror_dir) + end + def mirror_sync fork_info = @queue.pop mirror_dir = "/srv/git/#{fork_info['git_repo']}" mirror_dir = "#{mirror_dir}.git" unless fork_info['is_submodule'] - possible_new_refs = false - if File.directory?(mirror_dir) - possible_new_refs = git_fetch(mirror_dir) - else - possible_new_refs = git_clone(fork_info['url'], mirror_dir) - end + possible_new_refs = git_repo_download(fork_info['url'], mirror_dir) feedback(fork_info['git_repo'], possible_new_refs) end -- 2.23.0

2 1

[PATCH v2 compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'new_ref_times'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 38 +++++++++++++++++++++++++++++++ 2 files changed, 51 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..74eda85 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "source": [["x_params", $date1, ..., $date_today], +# ["new_ref_times", $times_1, ..., $times_today]] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..291fd9f 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,41 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'source' => [['x_params'], ['new_ref_times']] + } + dev = 1 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['source'][0][dev] = day_s + echart['source'][1][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci] lib/job.cr: add a new os_mount named local
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/job.cr | 2 +- src/scheduler/kernel_params.cr | 1 + 2 files changed, 2 insertions(+), 1 deletion(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 964c520..c013aa7 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -344,7 +344,7 @@ class Job end # defaults to the 1st value - VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container"] + VALID_OS_MOUNTS = ["initramfs", "nfs", "cifs", "container", "local"] private def set_os_mount if is_docker_job? diff --git a/src/scheduler/kernel_params.cr b/src/scheduler/kernel_params.cr index dea9215..ed64bea 100644 --- a/src/scheduler/kernel_params.cr +++ b/src/scheduler/kernel_params.cr @@ -18,6 +18,7 @@ class Job "cifs" => "root=cifs://#{OS_HTTP_HOST}#{os_real_path}" + ",guest,ro,hard,vers=1.0,noacl,nouser_xattr", "initramfs" => "rdinit=/sbin/init prompt_ramdisk=0", + "local" => "root=/dev/mapper/os-#{self.os}_#{self.os_arch}_#{self.os_version}", "container" => "", } -- 2.23.0

1 0

[PATCH compass-ci] container/web-backend: add api to query new_refs count as echart
by Li Yuanchao 01 Feb '21

01 Feb '21

Query the new_refs count data from es, and format it as echart needs, with 'x_params' and 'y_params'. Return it to webapi. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 13 +++++++++++ src/lib/web_backend.rb | 39 +++++++++++++++++++++++++++++++ 2 files changed, 52 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index ff8fcff..6f85f35 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -133,3 +133,16 @@ end get '/get_tbox_state' do get_tbox_state(params) end + +# GET /get_repo_statistics?git_repo=$git_repo +# Response: +# - { +# "title": "new refs statistics", +# "unit": "times", +# "x_name": "date", +# "x_params": [$date1, ..., $date_today], +# "y_params": [$times_1, ..., $times_today] +# } +get '/get_repo_statistics' do + new_refs_statistics(params) +end diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 61e3334..1c29c88 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -510,3 +510,42 @@ def get_tbox_state(params) end [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end + +def get_echart(statistics) + echart = { + 'title' => 'new refs statistics', + 'unit' => 'times', + 'x_name' => 'date', + 'x_params' => [], + 'y_params' => [] + } + dev = 0 + # The day will be 2021-01-01 + day = Date.new(2021, 1, 1) + today = Date.today + while day <= today + day_s = day.to_s + echart['x_params'][dev] = day_s + echart['y_params'][dev] = statistics[day_s] || 0 + day += 1 + dev += 1 + end + echart +end + +def query_repo_statistics(params) + query = { "query": { "match": { "_id": params[:git_repo] } } } + result = ES_CLIENT.search(index: 'repo', body: query)['hits'] + statistics = result['total'].positive? ? result['hits'][0]['_source']['new_refs_count'] : {} + get_echart(statistics) +end + +def new_refs_statistics(params) + begin + body = query_repo_statistics(params) + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'new refs statistics error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

2 2

[PATCH compass-ci 2/2] service/lifecycle: to process the machine that times out
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/lifecycle.cr | 51 ++++++++++++++++++++++++++++++++++++++ src/lifecycle/lifecycle.cr | 8 ++++++ 2 files changed, 59 insertions(+) diff --git a/src/lib/lifecycle.cr b/src/lib/lifecycle.cr index af5cd07..75a18ec 100644 --- a/src/lib/lifecycle.cr +++ b/src/lib/lifecycle.cr @@ -4,6 +4,7 @@ require "kemal" require "yaml" +require "./scheduler_api" require "./web_env" require "../scheduler/elasticsearch_client" @@ -14,6 +15,7 @@ class Lifecycle @es = Elasticsearch::Client.new @env = env @log = env.log.as(JSONLogger) + @scheduler_api = SchedulerAPI.new end def alive(version) @@ -36,4 +38,53 @@ class Lifecycle } @es.search("testbox", query) end + + def get_timeout_testbox(size) + # why use "now+8h" + # The time zone in the ES database is 0 + # add 8 hours to the current time and convert it to time zone 0 + query = { + "size" => size, + "query" =>{ + "range" => { + "deadline" => { + "lt" => "now+8h" + } + } + } + } + + results = @es.search("testbox", query) + results["hits"]["hits"] + end + + def process_timeout_testbox + size = @env.params.query["size"]? || 20 + results = get_timeout_testbox(size) + return results unless results.is_a?(JSON::Any) + + results.as_a.each do |result| + hostname = result["_id"].to_s + hash = result["_source"] + close_lifecycle(hostname, hash, "timeout") + end + end + + # 1.reboot testbox, do in the fature + # 2.update testbox info in es + # 3.close job + def close_lifecycle(hostname, hash, close_reason) + state = hash["state"]? == "rebooting" ? hash["state"] : "forcibly_rebooting" + testbox_update_info = { + "deadline" => nil, + "state" => state + } + @es.update_tbox(hostname, testbox_update_info) + + # If the value of state is rebooting, + # the job has been closed and don't need to close it again + return if state == "rebooting" + + @scheduler_api.close_job(hash["job_id"], close_reason).status_message + end end diff --git a/src/lifecycle/lifecycle.cr b/src/lifecycle/lifecycle.cr index 4154a27..13d36b7 100644 --- a/src/lifecycle/lifecycle.cr +++ b/src/lifecycle/lifecycle.cr @@ -29,4 +29,12 @@ module Cycle get "/get_running_testbox" do |env| env.lifecycle.get_running_testbox.to_json end + + # curl http://localhost:11311/process_timeout_testbox?size=10 + # size indicates the number of records to be processed at a time + get "/process_timeout_testbox" do |env| + env.lifecycle.process_timeout_testbox + + "done" + end end -- 2.23.0

1 0

[PATCH compass-ci 1/2] lib/scheduler_api.cr: added parameter "job_state"
by Wu Zhende 01 Feb '21

01 Feb '21

Signed-off-by: Wu Zhende <wuzhende666(a)163.com> --- src/lib/scheduler_api.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/scheduler_api.cr b/src/lib/scheduler_api.cr index b8dd514..0fdcd12 100644 --- a/src/lib/scheduler_api.cr +++ b/src/lib/scheduler_api.cr @@ -10,10 +10,14 @@ class SchedulerAPI @host = ENV.has_key?("SCHED_HOST") ? ENV["SCHED_HOST"] : "172.17.0.1" end - def close_job(job_id) + def close_job(job_id, job_state = nil) + url = "/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}" + url += "&job_state=#{job_state}" if job_state + client = HTTP::Client.new(@host, port: @port) - response = client.get("/~lkp/cgi-bin/lkp-post-run?job_id=#{job_id}") + response = client.get(url) client.close() + return response end end -- 2.23.0

1 0

[PATCH compass-ci] lib/params_group.rb: change group_key value parse format
by Wang Yong 01 Feb '21

01 Feb '21

[before] os=openeuler/os_arch=aarch64/os_version=20.03/pp.sysbench-cpu.cpu_max_prime=20000/pp.sysbench-cpu.nr_threads=96/pp.sysbench-cpu.runtime=300/tbox_group=taishan200-2280-2s48p-256g [after] os=openeuler os_arch=aarch64 os_version=20.03 pp.sysbench-cpu.cpu_max_prime=20000 pp.sysbench-cpu.nr_threads=96 pp.sysbench-cpu.runtime=300 tbox_group=taishan200-2280-2s48p-256g it's easy to paste value and pass these to es-find Signed-off-by: Wang Yong <wangyong0117(a)qq.com> --- lib/params_group.rb | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/lib/params_group.rb b/lib/params_group.rb index 9be2cbe..2082c5d 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -124,7 +124,7 @@ end def get_group_key(group_params) group_str = group_params.each.map do |k, v| "#{k}=#{v}" - end.sort!.join('/') + end.sort!.join(' ') return group_str end -- 2.23.0

2 1

[PATCH v3 compass-ci 4/4] container/submit: README file
by Luan Shengde 01 Feb '21

01 Feb '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/submit/README.md | 26 ++++++++++++++++++++++++++ 1 file changed, 26 insertions(+) create mode 100644 container/submit/README.md diff --git a/container/submit/README.md b/container/submit/README.md new file mode 100644 index 0000000..6772961 --- /dev/null +++ b/container/submit/README.md @@ -0,0 +1,26 @@ +# Install lkp-tests and its depends in container. +# With docker container, will suit most of the linux OSes. +# Avoid much repetitive testing work to adapt various OSes. + +Prepare work: + apply account + config default yaml + generate ssh keys + +Installation: + git clone https://gitee.com/wu_fengguang/compass-ci.git + + cd compass-ci/container/submit + ./build + + Add soft link for executable file 'submit' to $PATH: + + ln -s $PWD/submit /usr/bin/submit + +Usage: + You can directly use the command 'submit' to submit jobs. + It's the same as you install the lkp-tests at your local host. + + Example: + + submit -c -m testbox=vm-2p8g borrow-1h.yaml -- 2.23.0

1 0