- Compass-ci - mailweb.openeuler.org

[PATCH v2 compass-ci] sched: use user lkp src in set_pp_initrd
by Cao Xueliang 16 Oct '20

16 Oct '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 14 ++++++++++++-- 1 file changed, 12 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index effba9b..20afa65 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -6,6 +6,7 @@ require "yaml" require "any_merge" require "scheduler/constants.cr" +require "scheduler/jobfile_operate.cr" struct JSON::Any def []=(key : String, value : String) @@ -80,6 +81,7 @@ class Job access_key access_key_file lkp_initrd_user + user_lkp_src kernel_uri kernel_append_root kernel_params @@ -126,6 +128,7 @@ class Job private def set_defaults append_init_field() + set_user_lkp_src() set_os_dir() set_result_root() set_result_service() @@ -381,6 +384,13 @@ class Job self["kernel_params"] = " #{kernel_common_params()} #{kernel_append_root} #{kernel_console()}" end + private def set_user_lkp_src + lkp_arch_cgz = "#{SRV_INITRD}/lkp/#{lkp_initrd_user}/lkp-#{os_arch}.cgz" + raise "The #{lkp_arch_cgz} does not exist." unless File.exists?(lkp_arch_cgz) + + self["user_lkp_src"] = Jobfile::Operate.prepare_lkp_tests(lkp_initrd_user, os_arch) + end + private def set_pp_initrd initrd_deps_arr = Array(String).new initrd_pkg_arr = Array(String).new @@ -392,11 +402,11 @@ class Job deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{user_lkp_src}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end - if File.exists?("#{ENV["LKP_SRC"]}/pkg/#{program}") + if File.exists?("#{user_lkp_src}/pkg/#{program}") initrd_pkg_arr << "#{initrd_http_prefix}" + JobHelper.service_path(pkg_dest_file) end end -- 2.23.0

1 0

[PATCH compass-ci] sbin/es-account.sh update es-mapping
by Lu Weitao 16 Oct '20

16 Oct '20

modify some fields to make them more specific Signed-off-by: Lu Weitao <luweitaobe(a)163.com> --- sbin/es-account.sh | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/sbin/es-account.sh b/sbin/es-account.sh index 2b739d5..635f4fc 100755 --- a/sbin/es-account.sh +++ b/sbin/es-account.sh @@ -18,16 +18,16 @@ else "my_commit_url": { "type": "keyword" }, - "uuid": { + "my_uuid": { "type": "keyword" }, - "email": { + "my_email": { "type": "keyword" }, - "name": { + "my_name": { "type": "keyword" }, - "user": { + "my_login_name": { "type": "keyword" } } -- 2.23.0

1 0

[PATCH v2 compass-ci 3/4] container/send-mail: update Dockerfile
by Luan Shengde 16 Oct '20

16 Oct '20

Signed-off-by: Luan Shengde <luanshengde2(a)huawei.com> --- container/send-mail/Dockerfile | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/container/send-mail/Dockerfile b/container/send-mail/Dockerfile index 36b145c..0cf3eeb 100644 --- a/container/send-mail/Dockerfile +++ b/container/send-mail/Dockerfile @@ -11,9 +11,7 @@ RUN apt-get update && \ ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime && \ echo 'Asia/Shanghai' > /etc/timezone -EXPOSE 25 - COPY mail-post.rb /usr/local/bin/ COPY send-mail.rb /usr/local/bin/ -CMD ["/usr/local/bin/mail-post.rb"] +WORKDIR /usr/local/bin/ -- 2.23.0

1 0

[PATCH compass-ci 2/2] web-backend: add search feature to "/get_repos" api
by Zhang Yuhang 16 Oct '20

16 Oct '20

Signed-off-by: Zhang Yuhang <zhangyuhang25(a)huawei.com> --- container/web-backend/web-backend | 6 ++- src/lib/web_backend.rb | 82 ++++++++++++++++--------------- 2 files changed, 48 insertions(+), 40 deletions(-) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index 60c7827..fcc765f 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -9,6 +9,7 @@ require "#{CCI_SRC}/src/lib/web_backend.rb" set :bind, '0.0.0.0' set :port, 32767 set :show_exceptions, false +set :environment, :production # GET /compare_candidates # return: {"query_conditions": {...}, "dimension": [...]} @@ -48,7 +49,10 @@ get '/get_jobs' do get_jobs(params) end -# GET /get_repos?page_size=12&page_num=3 +# GET /get_repos?git_repo=a&page_size=12&page_num=3 +# git_repo: +# > optional +# # page_size: # > optional # - default: 20 diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index 31aa0c3..48cfadb 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -95,7 +95,7 @@ def compare_candidates_body OS: [ { os: 'openeuler', os_version: ['1.0', '20.03'] }, { os: 'centos', os_version: ['7.6', '7.8', '8.1', 'sid'] }, - { os: 'debian', os_version: ['10', 'sid'] }, + { os: 'debian', os_version: %w[10 sid] }, { os: 'archlinux', os_version: ['5.5.0-1'] } ], os_arch: %w[aarch64 x86_64], @@ -304,7 +304,7 @@ def get_jobs_body(params) banner: get_banner(git_repo, branches), jobs: jobs, fields: FIELDS - } + }.to_json end def get_jobs(params) @@ -314,66 +314,70 @@ def get_jobs(params) warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get jobs error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end -def get_repo_url(repo_file) - return unless File.file? repo_file +def get_repo_url(urls) + return unless urls.is_a?(Array) - urls = YAML.load_file(repo_file)['url'] urls.each do |url| return url if url[0, 4] == 'http' end urls[0] end -def get_repo(git_repo, repo_file = nil) - repo = { - git_repo: git_repo, - git_url: nil - } - return repo if !git_repo && !repo_file - - if !git_repo - git_repo = repo_file[UPSTREAM_REPOS_PATH.size + 1, repo_file.size - 1] - repo[:git_repo] = git_repo - elsif !repo_file - repo_file = File.join(UPSTREAM_REPOS_PATH, git_repo) +def get_repo(git_repo) + repo = nil + if git_repo + must = [{ regexp: { git_repo: ".*#{git_repo}.*" } }] + repo = query_repos(must, from: 0, size: 1)[0] end - - repo[:git_url] = get_repo_url(repo_file) - repo + repo || {} end -def repo_files_list - Dir["#{UPSTREAM_REPOS_PATH}/*/*/*"].sort +def query_repos(must, from: 0, size: 1) + query = { + query: { bool: { must: must } }, + size: size, + from: from, + sort: [{ + git_repo: { order: 'asc' } + }] + } + result = ES_CLIENT.search index: 'repo', body: query + repos = [] + result['hits']['hits'].each do |r| + r = r['_source'] + repos << { + git_url: get_repo_url(r['url']), + git_repo: r['git_repo'] + } + end + repos end -def get_repos_list(repo_files, from, finish, total) - repos_list = [] - - total.times do |index| - next if index < from - break if index >= finish +def search_repos(git_repo, page_size, page_num) + size = page_size + from = size * page_num + must = git_repo ? [{ regexp: { git_repo: ".*#{git_repo}.*" } }] : [] + count_query = { query: { bool: { must: must } } } + total = ES_CLIENT.count(index: 'repo', body: count_query)['count'] + return [], total if wrong_size?(size, from) - repos_list << get_repo(nil, repo_files[index]) - end - repos_list + return query_repos(must, from: from, size: size), total end def get_repos_body(params) - repo_files = repo_files_list page_size = get_positive_number(params[:page_size], 20) page_num = get_positive_number(params[:page_num], 1) - 1 + git_repo = params[:git_repo] - from = page_num * page_size - finish = from + page_size - total = repo_files.size + repos, total = search_repos(git_repo, page_size, page_num) { total: total, - repos: get_repos_list(repo_files, from, finish, total) - } + repos: repos + }.to_json end def get_repos(params) @@ -383,5 +387,5 @@ def get_repos(params) warn e.message return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] end - [200, headers.merge('Access-Control-Allow-Origin' => '*'), JSON.dump(body)] + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] end -- 2.23.0

2 2

[PATCH v3 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..984b9a1 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) -- 2.23.0

1 0

[PATCH v2 compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 19 ++++++++++--------- 1 file changed, 10 insertions(+), 9 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7785c2a..a0cae68 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + unless hash_dup[key] == @hash[key] + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,9 +389,9 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" - if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") + if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}")) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var osimage_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

osimage_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fd6e79b 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + osimage_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{osimage_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{osimage_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

1 0

[PATCH compass-ci] job.cr: def var initrd_os_dir for used multi times
by Xu Xijian 16 Oct '20

16 Oct '20

initrd_os_dir is used twice here, define a var fo it. Signed-off-by: Xu Xijian <xuxijian(a)huawei.com> --- src/lib/job.cr | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..fc712ba 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -299,10 +299,11 @@ class Job private def initramfs_initrds temp_initrds = [] of String + initrd_os_dir = "#{SRV_INITRD}/osimage/#{os_dir}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/current")}" + "#{JobHelper.service_path("#{initrd_os_dir}/current")}" temp_initrds << "#{INITRD_HTTP_PREFIX}" + - "#{JobHelper.service_path("#{SRV_INITRD}/osimage/#{os_dir}/run-ipconfig.cgz")}" + "#{JobHelper.service_path("#{initrd_os_dir}/run-ipconfig.cgz")}" temp_initrds.concat(initrd_deps.split(/ /)) unless initrd_deps.empty? temp_initrds.concat(initrd_pkg.split(/ /)) unless initrd_pkg.empty? -- 2.23.0

2 2

[PATCH lkp-tests] fix(iso2rootfs): modify the soft link name
by Yu Chuan 16 Oct '20

16 Oct '20

'dailybuild' is closer to logic than 'debug'. Signed-off-by: Yu Chuan <13186087857(a)163.com> --- tests/iso2rootfs | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/tests/iso2rootfs b/tests/iso2rootfs index 7ab4a08b0d60..524ab6695f93 100755 --- a/tests/iso2rootfs +++ b/tests/iso2rootfs @@ -72,7 +72,7 @@ get_daily_iso_checksum() mount_rootfs() { - DEBUG_ROOTFS_DIR="debug-versions/dailybuild" + DAILYBUILD_ROOTFS_DIR="debug-versions/dailybuild" case ${rootfs_protocol} in "nfs") ROOTFS_SERVER_PATH="${rootfs_server}:${rootfs_path}/${iso_os}/${iso_arch}" @@ -103,7 +103,7 @@ get_cache_iso_checksum() { mount_rootfs - CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-newest.sha256sum" + CHECKSUM_FILE_CACHE="${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-newest.sha256sum" [ ! -f "${CHECKSUM_FILE_CACHE}" ] || SHA256SUM_CACHE=$(awk '{print $1}' "$CHECKSUM_FILE_CACHE") } @@ -255,7 +255,7 @@ download_compass_ci() config_rootfs_dir() { - ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DEBUG_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") + ROOTFS_DES_DIR=${ROOTFS_LOCAL_PATH}/${DAILYBUILD_ROOTFS_DIR}/${iso_version}-$(date "+%Y%m%d%H%M%S") [ -d "${ROOTFS_DES_DIR}" ] && ROOTFS_DES_DIR="${ROOTFS_DES_DIR}-${HOSTNAME##*--}" mkdir -p "$ROOTFS_DES_DIR" @@ -310,10 +310,10 @@ test_rootfs() local root_path case ${rootfs_protocol} in "nfs") - root_path="${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" + root_path="${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}")" ;; "cifs") - root_path="cifs:${ROOTFS_SERVER_PATH}/${DEBUG_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" + root_path="cifs:${ROOTFS_SERVER_PATH}/${DAILYBUILD_ROOTFS_DIR}/$(basename "${ROOTFS_DES_DIR}"),${ROOTFS_MOUNT_PARAM}" ;; *) die "rootfs_protocol is none, exit !!!" @@ -363,10 +363,11 @@ post_works() mv "${CHECKSUM_FILE_CACHE}.tmp" "${CHECKSUM_FILE_CACHE}" + local soft_link="${iso_version}-dailybuild" cd "${ROOTFS_LOCAL_PATH}" && - rm -f "${iso_version}-debug" && - ln -s "${DEBUG_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ - "${iso_version}-debug" + rm -f "${soft_link}" && + ln -s "${DAILYBUILD_ROOTFS_DIR}/$(basename ${ROOTFS_DES_DIR})" \ + "${soft_link}" cd / && umount "${ROOTFS_LOCAL_PATH}" -- 2.23.0

2 2

[PATCH compass-ci] fix: scheduler: remove log for same id update
by Tong Qunfeng 16 Oct '20

16 Oct '20

when use job.update with same id, that's equal no update to id. remove the log "Should not direct update id, use update_id, ignore this" and fix another 1 crystal-format, 1 log message. Signed-off-by: Tong Qunfeng <taxcom(a)tom.com> --- src/lib/job.cr | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 32aa662..ffe3749 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -108,12 +108,13 @@ class Job def update(hash : Hash) hash_dup = hash.dup - if hash_dup.has_key?("id") - hash_dup.delete("id") - puts "Should not direct update id, use update_id, ignore this" - end - if hash_dup.has_key?("tbox_group") - raise "Should not direct update tbox_group, use update_tbox_group" + ["id", "tbox_group"].each do |key| + if hash_dup.has_key?(key) + if "#{hash_dup[key]}" != "#{@hash[key]}" + raise "Should not direct update #{key}, use update_#{key}" + end + hash_dup.delete(key) + end end @hash.any_merge!(hash_dup) @@ -205,7 +206,7 @@ class Job def []=(key : String, value : String | Nil) if key == "id" || key == "tbox_group" - raise "Should not []= id and tbox_group, use update_#{key}" + raise "Should not use []= update #{key}, use update_#{key}" end @hash[key] = JSON::Any.new(value) if value end @@ -388,7 +389,7 @@ class Job program_params = @hash["pp"].as_h program_params.keys.each do |program| deps_dest_file = "#{SRV_INITRD}/deps/#{mount_type}/#{os_dir}/#{program}.cgz" - pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" + pkg_dest_file = "#{SRV_INITRD}/pkg/#{mount_type}/#{os_dir}/#{program}.cgz" if File.exists?("#{ENV["LKP_SRC"]}/distro/depends/#{program}") && File.exists?(deps_dest_file) initrd_deps_arr << "#{initrd_http_prefix}" + JobHelper.service_path(deps_dest_file) end -- 2.23.0

2 2