- Compass-ci - mailweb.openeuler.org

[PATCH compass-ci 2/2] src/lib/web_backend.rb: add methods to search testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

Query all testboxes from es, and return a non repetitive list Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- src/lib/web_backend.rb | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb index e548d61..a897b57 100644 --- a/src/lib/web_backend.rb +++ b/src/lib/web_backend.rb @@ -435,3 +435,31 @@ def template_body(request_body) formatter = FormatEchartData.new(compare_results, request_body) formatter.format_for_echart.to_json end + +def search_testboxes + query = { size: 0, aggs: { testboxes: { terms: { size: 10000, field: 'testbox' } } } } + buckets = es_query(query)['aggregations']['testboxes']['buckets'] + testboxes = [] + buckets.each_index do |index| + testboxes[index] = buckets[index]['key'] + end + return testboxes, testboxes.length +end + +def testboxes_body + testboxes, total = search_testboxes + { + total: total, + testboxes: testboxes + }.to_json +end + +def query_testboxes + begin + body = testboxes_body + rescue StandardError => e + warn e.message + return [500, headers.merge('Access-Control-Allow-Origin' => '*'), 'get repos error'] + end + [200, headers.merge('Access-Control-Allow-Origin' => '*'), body] +end -- 2.23.0

1 0

[PATCH compass-ci 1/2] container/web_backend: add a interface /get_testboxes
by Li Yuanchao 02 Dec '20

02 Dec '20

It would return a testbox list to web request Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- container/web-backend/web-backend | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/container/web-backend/web-backend b/container/web-backend/web-backend index d68e9f2..11ba604 100755 --- a/container/web-backend/web-backend +++ b/container/web-backend/web-backend @@ -103,3 +103,10 @@ post '/compare_template' do request.body.rewind # in case someone already read it compare_template(request.body.read) end + +# GET /get_testboxes +# Response: +# - { total: total, testboxes[ 'testbox1', 'testbox2', ... ] } +get '/get_testboxes' do + query_testboxes +end -- 2.23.0

1 0

[PATCH v4 compass-ci] doc/job: add arch-nr_node-nr_cpu-memory.md
by Zhang Yu 02 Dec '20

02 Dec '20

Add a document for introducing the arch, nr_node, nr_cpu, memory fields Signed-off-by: Zhang Yu <2134782174(a)qq.com> --- doc/job/arch-nr_node-nr_cpu-memory.md | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 doc/job/arch-nr_node-nr_cpu-memory.md diff --git a/doc/job/arch-nr_node-nr_cpu-memory.md b/doc/job/arch-nr_node-nr_cpu-memory.md new file mode 100644 index 0000000..c788607 --- /dev/null +++ b/doc/job/arch-nr_node-nr_cpu-memory.md @@ -0,0 +1,16 @@ +# arch, nr_node, nr_cpu, memory + +Meaning: +- Each test machine has some basic configurations, such as arch, nr_node, nr_cpu, memory + these fields mean: + arch: the architecture of the machine. + nr_node: number of NUMA nodes. + nr_cpu: number of logical CPUs. + memory: memory size of the machine. + +- These fields are from the file which in $LAB_SRC/hosts. +- Here is an example of these fields in the host file: + arch: aarch64 + nr_node: 4 + nr_cpu: 96 + memory: 256G -- 2.23.0

1 0

[PATCH compass-ci] lib/es_query.rb: add traverse es data of function
by Cui Yili 02 Dec '20

02 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- lib/es_query.rb | 18 ++++++++++++++++++ 1 file changed, 18 insertions(+) diff --git a/lib/es_query.rb b/lib/es_query.rb index c027aa9..4c99023 100644 --- a/lib/es_query.rb +++ b/lib/es_query.rb @@ -11,6 +11,7 @@ class ESQuery PORT = (ENV.key?('ES_PORT') ? ENV['ES_PORT'] : ES_PORT).to_i def initialize(host = HOST, port = PORT, index: 'jobs') @index = index + @scroll_id = "" @client = Elasticsearch::Client.new url: "http://#{host}:#{port}" raise 'Connect Elasticsearch error!' unless @client.ping end @@ -33,6 +34,23 @@ class ESQuery @client.search index: 'jobs*', body: query end + def traverse_field(size) + if @scroll_id.empty? + query = { + query: { + bool: { + must: {match_all: {} } + } + }, size: size + } + result = @client.search index: @index, scroll: '10m', body: query + @scroll_id = result['_scroll_id'] + return result + else + @client.scroll scroll: '10m', scroll_id: @scroll_id + end + end + def query_by_id(id) @client.get_source({ index: @index, type: '_doc', id: id }) rescue Elasticsearch::Transport::Transport::Errors::NotFound -- 2.23.0

1 0

[PATCH v3 lkp-tests 1/2] tests: replace *.tar.* address with local address
by Liu Shaofei 02 Dec '20

02 Dec '20

When the address of source field is the format of "*.tar.*", we need to replace it with local address. The following example: source=("https://github.com/azukiapp/${pkgname}/archive/v${pkgver}.tar.gz") => source=("http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/v${pkgver}.tar.gz") Signed-off-by: Liu Shaofei <liushaofei5(a)huawei.com> --- tests/build-pkg | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) diff --git a/tests/build-pkg b/tests/build-pkg index bdaef035..cde2f0ed 100755 --- a/tests/build-pkg +++ b/tests/build-pkg @@ -98,6 +98,26 @@ replace_source() sed -i "s|^source=[^)]*|$1|g" "$2" } +# replace *.tar.* address from source field in PKGBUILD +replace_tar_source() +{ + echo "source=(" > PKGBUILD.src + for url in ${source[@]} + do + echo "$url" | egrep '.+\.tar' && { + local_url="http://${LKP_SERVER}:8800/initrd/build-tar/${pkgname}/${url##*/}" + + # check whether local server exists *.tar.* file + wget -q --spider "${local_url}" + [ "$?" == 0 ] || continue + + url="\"${local_url}\"" + } + echo "$url" >> PKGBUILD.src + done + echo ")" >> PKGBUILD.src +} + build_source_pkg() { local repo_dir="" @@ -116,6 +136,8 @@ build_source_pkg() [ -n "$PKGBUILD_TAG" ] && replace_source "${upstream_source}" "PKGBUILD-$PKGBUILD_TAG" } + replace_tar_source + cgz_name="$PKG_MNT/${pack_to}/${pkgname}/${upstream_commit}.cgz" PACMAN=true BUILDDIR=$TMP CARCH=$os_arch PKGEXT=.cgz CGZDEST="$cgz_name" \ $LKP_SRC/sbin/makepkg -A --check --skippgpcheck --config $LKP_SRC/etc/makepkg.conf 2>&1 -- 2.23.0

2 2

[PATCH compass-ci] params_group.rb: delete $user if tbox_group has
by Li Yuanchao 02 Dec '20

02 Dec '20

Some tbox_groups have $user in es. Such as vm-2p8g--$user or vm-2p8g.$user. But their config is the same, they should be compared together. Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- lib/params_group.rb | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/lib/params_group.rb b/lib/params_group.rb index cde40af..6d5c47b 100644 --- a/lib/params_group.rb +++ b/lib/params_group.rb @@ -67,10 +67,17 @@ def filter_groups(groups) end end +def get_tbox_group(tbox_group) + tbox_group = tbox_group.split('--')[0] if tbox_group.include?('--') + tbox_group = tbox_group.split('.')[0] if tbox_group.include?('.') + return tbox_group +end + def get_all_params(job) all_params = {} job.each_key do |param| all_params[param] = job[param] if COMMON_PARAMS.include?(param) + all_params[param] = get_tbox_group(job[param]) if param == 'tbox_group' next unless param == 'pp' pp_params = get_pp_params(job[param]) -- 2.23.0

3 4

[PATCH compass-ci] sched: no need to check kernel config in initialized? function
by Cao Xueliang 02 Dec '20

02 Dec '20

Signed-off-by: Cao Xueliang <caoxl78320(a)163.com> --- src/lib/job.cr | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/src/lib/job.cr b/src/lib/job.cr index 7758e78..2a811cb 100644 --- a/src/lib/job.cr +++ b/src/lib/job.cr @@ -146,8 +146,8 @@ class Job set_result_root() set_result_service() set_os_mount() - set_kernel() set_depends_initrd() + set_kernel() set_initrds_uri() set_lkp_server() set_sshr_info() @@ -345,7 +345,11 @@ class Job "SCHED_HOST", "SCHED_PORT"] - initialized_keys -= ["my_uuid"] + initialized_keys -= ["my_uuid", + "kernel_version", + "kernel_uri", + "kernel_params", + "linux_vmlinuz_path"] initialized_keys.each do |key| return false unless @hash.has_key?(key) -- 2.23.0

1 0

[PATCH compass-ci] auto_submit.yaml: change git_repo
by Li Yuanchao 02 Dec '20

02 Dec '20

To keep style the same Signed-off-by: Li Yuanchao <lyc163mail(a)163.com> --- sbin/auto_submit.yaml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sbin/auto_submit.yaml b/sbin/auto_submit.yaml index f741b79..e0f52ba 100644 --- a/sbin/auto_submit.yaml +++ b/sbin/auto_submit.yaml @@ -1,6 +1,6 @@ -AvxToNeon/AvxToNeon: +a/AvxToNeon/AvxToNeon: - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 api-avx2neon.yaml -compass-ci/compass-ci: +c/compass-ci/compass-ci: - testbox=vm-2p8g os=openeuler os_version=20.03 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=centos os_version=7.6.1810 os_mount=initramfs os_arch=aarch64 deploy-cci.yaml - testbox=vm-2p8g os=debian os_version=sid os_mount=initramfs os_arch=aarch64 deploy-cci.yaml -- 2.23.0

1 0

[PATCH compass-ci] result-webdav/start: add /etc/localtime
by Cui Yili 02 Dec '20

02 Dec '20

Signed-off-by: cuiyili <2268260388(a)qq.com> --- container/result-webdav/start | 1 + 1 file changed, 1 insertion(+) diff --git a/container/result-webdav/start b/container/result-webdav/start index 01141c3..d7b1097 100755 --- a/container/result-webdav/start +++ b/container/result-webdav/start @@ -13,6 +13,7 @@ cmd=( --restart=always -p 3080:3080 -v $PWD/nginx.conf:/usr/local/openresty/nginx/conf/nginx.conf + -v /etc/localtime:/etc/localtime:ro -v /srv/result/:/srv/result/ -v /srv/initrd/:/srv/initrd/ result-webdav -- 2.23.0

1 0

[PATCH compass-ci] answerback-email: add execute tool limit
by Luan Shengde 02 Dec '20

02 Dec '20

limit the scope of executing answerback-email to 'z9' [why] when call es_client to store the account data, the data will be stored in the es installed at the localhost. It will cause validation to fail when submit a job if the data is not at the server 'z9'. Signed-off-by: Luan Shengde <shdluan(a)163.com> --- container/assign-account/answerback-email.rb | 33 +++++++++++++++++--- 1 file changed, 29 insertions(+), 4 deletions(-) diff --git a/container/assign-account/answerback-email.rb b/container/assign-account/answerback-email.rb index 825d815..77a2c1f 100755 --- a/container/assign-account/answerback-email.rb +++ b/container/assign-account/answerback-email.rb @@ -187,11 +187,14 @@ def apply_account(my_info, conf_info) end def check_my_email(my_info) + return true if my_info['my_email'] + message = "No email address specified\n" message += "use -e to add an email address for applying account\n" message += 'or use -f to add an email file' + puts message - raise message if my_info['my_email'].nil? + return false end def build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) @@ -219,13 +222,34 @@ def build_my_info_from_account_info(my_info, account_info, conf_info) my_info['my_login_name'] = account_info['my_login_name'] unless conf_info['is_update_account'] end +def check_server + return true if ENV['HOSTNAME'] == 'crystal.ci' + + message = 'please run the tool on z9 server' + + puts message + return false +end + +def check_my_name_exist(my_info) + return true if my_info['my_name'] + + message = 'No my_name found, please use -n to add one' + puts message + + return false +end + def send_account(my_info, conf_info, email_info, my_info_es, stdin_info) - check_my_email(my_info) + puts 'check_erver' + return unless check_server + return unless check_my_email(my_info) + my_info['my_uuid'] = %x(uuidgen).chomp unless conf_info['is_update_account'] build_my_info_from_input(my_info, email_info, my_info_es, stdin_info) - message = 'No my_name found, please use -n to add one' - raise message if my_info['my_name'].nil? + puts 'check_my_name' + return unless check_my_name_exist(my_info) account_info = apply_account(my_info, conf_info) build_my_info_from_account_info(my_info, account_info, conf_info) @@ -241,6 +265,7 @@ def send_mail(my_info, account_info, conf_info) else build_message(my_info['my_email'], account_info) end + %x(curl -XPOST '#{SEND_MAIL_HOST}:#{SEND_MAIL_PORT}/send_mail_text' -d "#{message}") end -- 2.23.0

1 1