February 2022 - Compass-ci - mailweb.openeuler.org

[PATCH private-cci] patch for test
by luanshengde＠compass-ci.org 28 Feb '22

28 Feb '22

From: Luan Shengde <shdluan(a)163.com> Signed-off-by: Luan Shengde <shdluan(a)163.com> --- bbb | 1 + 1 file changed, 1 insertion(+) create mode 100644 bbb diff --git a/bbb b/bbb new file mode 100644 index 0000000..d670460 --- /dev/null +++ b/bbb @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] for test gmail
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: Luan Shengde <shdluan(a)163.com> Signed-off-by: Luan Shengde <shdluan(a)163.com> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] for test gmail
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: Luan Shengde <shdluan(a)163.com> Signed-off-by: Luan Shengde <shdluan(a)163.com> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] test patch
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: Luan Shengde <shdluan(a)163.com> test patch log Signed-off-by: Luan Shengde <shdluan(a)163.com> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] test patch
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: Luan Shengde <shdluan(a)163.com> test patch log Signed-off-by: Luan Shengde <shdluan(a)163.com> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] test patch
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: Luan Shengde <shdluan(a)163.com> test patch log Signed-off-by: Luan Shengde <shdluan(a)163.com> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] test patch
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: Luan Shengde <shdluan(a)163.com> test patch log Signed-off-by: Luan Shengde <shdluan(a)163.com> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] for test gmail
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: luanshd <luanshengde(a)compass-ci.org> Signed-off-by: luanshd <luanshengde(a)compass-ci.org> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

[PATCH compass-ci] for test gmail
by luanshengde＠compass-ci.org 26 Feb '22

26 Feb '22

From: luanshd <luanshengde(a)compass-ci.org> Signed-off-by: luanshd <luanshengde(a)compass-ci.org> --- aaa | 1 + 1 file changed, 1 insertion(+) create mode 100644 aaa diff --git a/aaa b/aaa new file mode 100644 index 00000000..d670460b --- /dev/null +++ b/aaa @@ -0,0 +1 @@ +test content -- 2.23.0

1 0

meeting
by Lu Weitao 24 Feb '22

24 Feb '22

aa

1 0

test mail 2
by weijihui 24 Feb '22

24 Feb '22

test msg

1 0

meeting
by Lu Weitao 22 Feb '22

22 Feb '22

aa

1 0

id
by Lu Weitao 22 Feb '22

22 Feb '22

aa

1 0

key
by Luan Shengde 21 Feb '22

21 Feb '22

1 0

test
by Luan Shengde 21 Feb '22

21 Feb '22

1 0

test mail
by Luan Shengde 21 Feb '22

21 Feb '22

1 0

Re: [PATCH v6 lkp-tests] sbin/search, sbin/select: add es search cmd
by Wu Fengguang 17 Feb '22

17 Feb '22

On Thu, Feb 17, 2022 at 02:45:04PM +0800, Wei Jihui wrote: >On Thu, Feb 17, 2022 at 12:44:43PM +0800, Wu Fengguang wrote: >>On Wed, Feb 16, 2022 at 09:10:34PM +0800, Wei Jihui wrote: >>>sbin/search is cci client cmd for query es by dsl >> >>copy/paste error > >what is the error? That's not related to SQL. The below is. >>>sbin/select is cci client cmd for query es by sql Thanks, Fengguang

1 0

test mail
by Cao Xueliang 16 Feb '22

16 Feb '22

test

1 0

test reply 8
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test reply 7
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test reply 6
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test auto reply 5
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test auto reply 4
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test mail for auto reply 3
by Luan Shengde 15 Feb '22

15 Feb '22

test mail for auto reply 3

1 0

for test auto reply
by Luan Shengde 15 Feb '22

15 Feb '22

test msg for auto reply

1 0

test mail for compass-ci.org
by Cao Xueliang 15 Feb '22

15 Feb '22

test msg

1 0

Re: [PATCH v4 compass-ci] data-api: add es search api
by weijihui 15 Feb '22

15 Feb '22

On Tue, Feb 15, 2022 at 09:35:52AM +0800, Wei Jihui wrote: >On Mon, Feb 14, 2022 at 07:21:33PM +0800, Wu Fengguang wrote: >>On Mon, Feb 14, 2022 at 06:25:41PM +0800, weijihui wrote: >>>On Fri, Feb 11, 2022 at 06:51:34PM +0800, Wu Fengguang wrote: >>>>On Thu, Feb 10, 2022 at 09:25:37PM +0800, Wei Jihui wrote: >>>>>From: Wei Jihui <weijihuiall(a)163.com> >>>>> >>>>>1. /data_api/es/:index/_search is es search by dsl >>>> >>>>>2. /data_api/es/:index/_search_by_sql is es search by sql >>>> >>>>建议用 >>>> /data_api/_opendistro/_sql >>> >>>但是现在用的不是这个接口，用的是https://github.com/NLPchina/elasticsearch-sql >> >>两个repo有继承性，基本是兼容的。 >>要么也可以用NLPchina的接口，怎么也比现在的这个好: >> >> 2. /data_api/es/:index/_search_by_sql is es search by sql > >好的，那就用这个/data_api/es/:index/_nlpcn/sql 指向原始的接口 /data_api/_nlpcn/sql sinatra match pattern /data_api/es/:index/_search sinatra match pattern /data_api/redis/xxx sinatra match pattern Thanks, Fengguang

1 0

Re: [todo] auto submit cci-makepkg for all tests
by Wu Fengguang 15 Feb '22

15 Feb '22

On Mon, Feb 14, 2022 at 04:34:20PM +0800, Cao Xueliang wrote: ># auto submit cci-makepkg before run test suite > >example: >- job.yaml >--- >'#! rpmbuild.yaml': >suite: rpmbuild >category: functional >rpmbuild: >repo_addr: https://vault.centos.org/6.10/os/Source/SPackages/ed-1.1-3.3.el6.src.rpm >arch: aarch64 >custom_repo_name: unstable/compatible/6 >mount_repo_addr: https://api.compass-ci.openeuler.org:20018/rpm/testing/openeuler-20.03-LTS-… >1/compat-centos6/standard_\$basearch/\$basearch >mount_repo_name: compatible-6 >docker_image: openeuler:20.03-LTS-SP1 >``` >- generate cci-makepkg.yaml ># detect $LKP_SRC/pkg/rpmbuild/PKGBUILD first ># detect /srv/initrd/pkg/container/openeuler/aarch64/20.03-LTS-SP1/rpmbuild/latest.cgz >``` >suite: cci-makepkg >category: functional > >cci-makepkg: > benchmark: rpmbuild # from rpmbuild.yaml >os: # from rpmbuild.yaml >os_arch: # from rpmbuild.yaml >os_mount: # from rpmbuild.yaml >``` Looks good. >- submit cci-makepkg.yaml & rpmbuild.yaml > - if cci-makepkg job success and exists latest.cgz then submit rpmbuild.yaml > - if cci-makepkg job not success then echo error & no need submit rpmbuild.yaml Just to make it clear: Here "submit cci-makepkg.yaml" should not mean "cci submit". The scheduler should - check and setup job depends - auto create and dispatch cci-makepkg.yaml - continue dispatch rpmbuild.yaml, when the above job succeeded Thanks, Fengguang

1 0

Re: [PATCH v5 compass-ci] data-api: add es search api
by Wei Jihui 15 Feb '22

15 Feb '22

On Mon, Feb 14, 2022 at 07:07:14PM +0800, Wei Jihui wrote: >From: Wei Jihui <weijihuiall(a)163.com> > pls ignore this patch. Thanks, Jihui >1. /data_api/es/:index/_search is es search by dsl >2. /data_api/es/:index/_search_by_sql is es search by sql > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > container/data-api/data-api | 36 +++++++----- > lib/es_client.rb | 13 ++-- > src/lib/data_api.rb | 93 +++++------------------------ > src/lib/data_api/es_data_api.rb | 101 ++++++++++++++++++++++++++++++++ > 4 files changed, 143 insertions(+), 100 deletions(-) > create mode 100644 src/lib/data_api/es_data_api.rb > >diff --git a/container/data-api/data-api b/container/data-api/data-api >index 2bd6e527..834ec702 100755 >--- a/container/data-api/data-api >+++ b/container/data-api/data-api >@@ -7,8 +7,6 @@ require 'sinatra' > > CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' > require "#{CCI_SRC}/src/lib/data_api.rb" >-require "#{CCI_SRC}/src/lib/service_logs/service_logs.rb" >-require "#{CCI_SRC}/lib/es_query.rb" > > set :bind, '0.0.0.0' > set :port, 10005 >@@ -17,24 +15,32 @@ set :environment, :production > > # POST > # eg1: >-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >+# curl -X POST localhost:10005/data_api/es/jobs/_search -H 'Content-Type: application/json' -d '{ > # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >-# "index": "jobs", >-# "my_token": "16132550-...", >-# "my_account": "auto-submit", >-# "query_type": "eql" # can be omitted in this scenario >+# "cci_credentials": { >+# "my_token": "16132550-...", >+# "my_account": "auto-submit" >+# } > # }' >-# eg2: >-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >+# Response: >+# - es_result : JSON >+post '/data_api/es/:index/_search' do >+ request.body.rewind # in case someone already read it >+ es_search(params['index'], request.body.read) >+end >+ >+# POST >+# eg1: >+# curl -X POST localhost:10005/data_api/es/jobs/_search_by_sql -H 'Content-Type: application/json' -d '{ > # "query": "SELECT * FROM jobs WHERE ...", >-# "index": "jobs", >-# "my_token": "16132550-...", >-# "my_account": "auto-submit", >-# "query_type": "sql" # required >+# "cci_credentials": { >+# "my_token": "16132550-...", >+# "my_account": "auto-submit" >+# } > # }' > # Response: > # - es_result : JSON >-post '/data_api/es_find' do >+post '/data_api/es/:index/_search_by_sql' do > request.body.rewind # in case someone already read it >- es_find(request.body.read) >+ es_search_by_sql(params['index'], request.body.read) > end >diff --git a/lib/es_client.rb b/lib/es_client.rb >index 4ae1fc00..63e42d76 100644 >--- a/lib/es_client.rb >+++ b/lib/es_client.rb >@@ -6,9 +6,9 @@ require_relative 'es_query.rb' > # ------------------------------------------------------------------------------------------- > # put_source_by_id(source_id, source_content) > # - put a source to ES /<target>/_doc/<_id>, ingore the existence of source >-# >-# query_by_sql(query_sql) >-# - query es db by query_sql, query_sql is sql, like "SELECT * FROM JOBS WHERE ..." >+# >+# search_by_sql(search_sql) >+# - search es db by search_sql, search_sql is sql, like "SELECT * FROM JOBS WHERE ..." > # > # ------------------------------------------------------------------------------------------- > class ESClient < ESQuery >@@ -29,12 +29,11 @@ class ESClient < ESQuery > ) > end > >- # query es db by query_sql, query_sql is sql, like: >+ # search es db by search_sql, search_sql is sql, like: > # - "SELECT id, suite FROM JOBS" > # - "SELECT * FROM accounts WHERE my_account='test_user'" > # this plugin from: https://github.com/NLPchina/elasticsearch-sql >- def query_by_sql(query_sql) >- @client.perform_request('GET', '_nlpcn/sql', {}, query_sql) >+ def search_by_sql(search_sql) >+ @client.perform_request('GET', '_nlpcn/sql', {}, search_sql) > end >- > end >diff --git a/src/lib/data_api.rb b/src/lib/data_api.rb >index eeaf0f98..8aa4304b 100644 >--- a/src/lib/data_api.rb >+++ b/src/lib/data_api.rb >@@ -3,90 +3,27 @@ > # frozen_string_literal: true > > CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' >-require 'json' >-require 'elasticsearch' >-require 'set' >-require_relative '../../lib/constants.rb' >-require_relative '../../lib/es_query.rb' >-require_relative '../../lib/es_client.rb' >-require_relative '../../lib/json_logger.rb' >+require "#{CCI_SRC}/lib/json_logger.rb" >+require "#{CCI_SRC}/src/lib/data_api/es_data_api.rb" > >-ES_ACCOUNTS = ESQuery.new(index: 'accounts') >-REQUIRED_TOKEN_INDEX = Set.new(['jobs']) >-UNOPEN_INDEX = Set.new(['accounts']) >-ES_QUERY_KEYWORD = Set.new(['term', 'match']) >- >-def es_find(params) >+def es_search(index, params) > begin >- result = query_es(params) >+ result = EsDataApi.search(index, params) > rescue StandardError => e >- log_error({ >- 'message' => e.message, >- 'error_message' => "query es error" >- }) >- return [500, headers.merge('Access-Control-Allow-Origin' => '*'), e.message] >+ error_msg = { 'error_msg' => e.message } >+ log_error(error_msg) >+ return [200, headers.merge('Access-Control-Allow-Origin' => '*'), error_msg.to_json] > end > [200, headers.merge('Access-Control-Allow-Origin' => '*'), result.to_json] > end > >-def query_es(params) >- request_body = JSON.parse(params) >- index = request_body['index'] || 'jobs' >- query = request_body['query'] || {'query' => {}} >- return "#{index} is not opened" if UNOPEN_INDEX.any? { |unopen_index| index.include?(unopen_index) } >- >- if REQUIRED_TOKEN_INDEX.any? { |req_index| index.include?(req_index) } >- my_account = request_body['my_account'] >- my_token = request_body['my_token'] >- return "missed my_account" unless my_account >- return "missed my_token" unless my_token >- return "user authentication failed" unless verify_user(my_account, my_token) >- >- query = build_account_query(query, my_account, request_body['query_type'], index) >- end >- >- if request_body['query_type'] == 'sql' >- es = ESClient.new(index: index) >- return es.query_by_sql(query).body >- else >- es = Elasticsearch::Client.new(hosts: ES_HOSTS) >- return es.search index: index + '*', body: query >- end >-end >- >-def verify_user(my_account, my_token) >- query = {} >- query['my_account'] = my_account if my_account >- query['my_token'] = my_token if my_token >- >- result = ES_ACCOUNTS.multi_field_query(query)['hits']['hits'] >- return nil if result.size == 0 >- >- result[0]['_source']['my_token'] == my_token >-end >- >-def build_account_query(query, my_account, query_type, index) >- if query_type == 'sql' >- user_limit = "my_account='#{my_account}'" >- if query =~ /where/i >- query = query.gsub(/FROM\s*\S+/i, "FROM #{index} ") >- query = query.gsub(/where/i, "WHERE #{user_limit} AND") >- else >- query = query.gsub(/from\s*\S+/i, "FROM #{index} WHERE #{user_limit}") >- end >- else >- query['query'] ||= {} >- query['query']['bool'] ||= {} >- query['query']['bool']['must'] ||= [] >- query['query']['bool']['must'] << {'term' => {'my_account' => my_account}} >- >- query['query'].each do |k, v| >- if ES_QUERY_KEYWORD.include?(k) >- query['query']['bool']['must'] << {k => v} >- query['query'].delete(k) >- end >- end >+def es_search_by_sql(index, params) >+ begin >+ result = EsDataApi.search_by_sql(index, params) >+ rescue StandardError => e >+ error_msg = { 'error_msg' => e.message } >+ log_error(error_msg) >+ return [200, headers.merge('Access-Control-Allow-Origin' => '*'), error_msg.to_json] > end >- >- query >+ [200, headers.merge('Access-Control-Allow-Origin' => '*'), result.to_json] > end >diff --git a/src/lib/data_api/es_data_api.rb b/src/lib/data_api/es_data_api.rb >new file mode 100644 >index 00000000..ed89b596 >--- /dev/null >+++ b/src/lib/data_api/es_data_api.rb >@@ -0,0 +1,101 @@ >+# SPDX-License-Identifier: MulanPSL-2.0+ >+# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >+# frozen_string_literal: true >+ >+CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' >+require 'json' >+require 'set' >+require "#{CCI_SRC}/lib/constants.rb" >+require "#{CCI_SRC}/lib/es_client.rb" >+ >+# this module is for es api >+# - search >+# - search_by_sql >+module EsDataApi >+ ES_ACCOUNTS = ESClient.new(index: 'accounts') >+ OPEN_INDEX = Set.new(['jobs']) >+ REQUIRED_TOKEN_INDEX = Set.new(['jobs']) >+ ES_QUERY_KEYWORD = Set.new(%w[term match]) >+ >+ def self.credentials_for_dsl(query, my_account) >+ query['query'] ||= {} >+ query['query']['bool'] ||= {} >+ query['query']['bool']['must'] ||= [] >+ query['query']['bool']['must'] << { 'term' => { 'my_account' => my_account } } >+ query = handle_dsl_query(query) >+ return query >+ end >+ >+ def self.handle_dsl_query(query) >+ query['query'].each do |k, v| >+ if ES_QUERY_KEYWORD.include?(k) >+ query['query']['bool']['must'] << { k => v } >+ query['query'].delete(k) >+ end >+ end >+ return query >+ end >+ >+ def self.check_my_account(request_body) >+ cci_credentials = request_body['cci_credentials'] || {} >+ my_account = cci_credentials['my_account'] >+ my_token = cci_credentials['my_token'] >+ raise 'user authentication failed, please check my_account and my_token.' unless verify_user(my_account, my_token) >+ >+ return my_account >+ end >+ >+ def self.search(index, params) >+ request_body = JSON.parse(params) >+ query = request_body['query'] || { 'query' => {} } >+ raise "#{index} is not opened for user query" unless OPEN_INDEX.include?(index) >+ >+ if REQUIRED_TOKEN_INDEX.include?(index) >+ my_account = check_my_account(request_body) >+ query = credentials_for_dsl(query, my_account) >+ end >+ es = Elasticsearch::Client.new(hosts: ES_HOSTS) >+ return es.search index: index + '*', body: query >+ end >+ >+ def self.credentials_for_sql(query, my_account, index) >+ user_limit = "my_account='#{my_account}'" >+ if query =~ /where/i >+ query = query.gsub(/FROM\s*\S+/i, "FROM #{index} ") >+ query = query.gsub(/where/i, "WHERE #{user_limit} AND") >+ else >+ query = query.gsub(/from\s*\S+/i, "FROM #{index} WHERE #{user_limit}") >+ end >+ return query >+ end >+ >+ def self.search_by_sql(index, params) >+ request_body = JSON.parse(params) >+ query = request_body['query'] >+ raise "#{index} is not opened for user query" unless OPEN_INDEX.include?(index) >+ >+ if REQUIRED_TOKEN_INDEX.include?(index) >+ my_account = check_my_account(request_body) >+ query = credentials_for_sql(query, my_account, index) >+ end >+ >+ es = ESClient.new(index: index) >+ p query >+ p 1 >+ return es.search_by_sql(query).body >+ end >+ >+ def self.verify_user(my_account, my_token) >+ raise 'missed my_account' unless my_account >+ raise 'missed my_token' unless my_token >+ >+ query = {} >+ query['my_account'] = my_account >+ query['my_token'] = my_token >+ >+ result = ES_ACCOUNTS.multi_field_query(query)['hits']['hits'] >+ return nil if result.empty? >+ >+ result[0]['_source']['my_token'] == my_token >+ end >+end >-- >2.23.0 >

1 0

Re: [PATCH v4 compass-ci] data-api: add es search api
by Wu Fengguang 15 Feb '22

15 Feb '22

On Mon, Feb 14, 2022 at 06:25:41PM +0800, weijihui wrote: >On Fri, Feb 11, 2022 at 06:51:34PM +0800, Wu Fengguang wrote: >>On Thu, Feb 10, 2022 at 09:25:37PM +0800, Wei Jihui wrote: >>>From: Wei Jihui <weijihuiall(a)163.com> >>> >>>1. /data_api/es/:index/_search is es search by dsl >> >>>2. /data_api/es/:index/_search_by_sql is es search by sql >> >>建议用 >> /data_api/_opendistro/_sql > >但是现在用的不是这个接口，用的是https://github.com/NLPchina/elasticsearch-sql 两个repo有继承性，基本是兼容的。要么也可以用NLPchina的接口，怎么也比现在的这个好: 2. /data_api/es/:index/_search_by_sql is es search by sql Thanks, Fengguang

2 1

Re: [PATCH v4 compass-ci] data-api: add es search api
by weijihui 14 Feb '22

14 Feb '22

On Fri, Feb 11, 2022 at 06:53:02PM +0800, Wu Fengguang wrote: >>+# "cci_authorization": { >>+# "my_token": "16132550-...", >>+# "my_account": "auto-submit" >>+# } > >>+# "cci_authorization": { >>+# "my_token": "16132550-...", >>+# "my_account": "auto-submit" >>+# } > >s/cci_authorization/cci_credentials/ > ok Thanks, Jihui >Thanks, >Fengguang

1 0

Re: [PATCH v4 compass-ci] data-api: add es search api
by weijihui 14 Feb '22

14 Feb '22

On Fri, Feb 11, 2022 at 06:51:34PM +0800, Wu Fengguang wrote: >On Thu, Feb 10, 2022 at 09:25:37PM +0800, Wei Jihui wrote: >>From: Wei Jihui <weijihuiall(a)163.com> >> >>1. /data_api/es/:index/_search is es search by dsl > >>2. /data_api/es/:index/_search_by_sql is es search by sql > >建议用 > /data_api/_opendistro/_sql 但是现在用的不是这个接口，用的是https://github.com/NLPchina/elasticsearch-sql Thanks, Jihui > >:index 有可能数据重复，此时不可信。 > >因为真正起作用的是 select 语句里的from， >必需解析后得到index，再基于它做认证。 > >Thanks, >Fengguang >

1 0

Re: [meeting] stats rules
by Wu Fengguang 14 Feb '22

14 Feb '22

># stats > > global stats pattern > > *.(log|message|element): > type: string - .log 出自下面的TODO - .log 命名不够清楚，是不是改成.logfile；类似的.message .element也可以改长一点 - .xxx 后缀有很小的可能与正常id混淆，改成#logfile#或者类似明显不同寻常的后缀，区分度较好，会比较安全和明显 Thanks, Fengguang --- [TODO] virttest stats TODO1 ===== (1/1) type_specific.io-github-autotest-libvirt.libvirt_usb_hotplug_device.attach_device.one_count.storage: FAIL: failed to attach device.\nDetail: command: '/bin/virsh attach-device avocado-vt-vm1 /tmp/lkp/xml_utils_temp_7xma5yyz.xml'\nexit_status: 1\nduration: 0.029788970947265625\ninterrupted: False\npid: 30074\nencoding: 'UTF-8'\nstdout: '\n'\nstderr: "error: Failed to a... (49.09 s) ==> libvirt_usb_hotplug_device.attach_device.one_count.storage.fail: 1 (for bisect) feature3.casexxx.fail: 1 TODO2 ===== for bisect report email feature3.casexxx.message => (1/1) type_specific.io-github-autotest-libvirt.libvirt_usb_hotplug_device.attach_device.one_count.storage: FAIL: failed to attach device.\nDetail: command: '/bin/virsh attach-device avocado-vt-vm1 /tmp/lkp/xml_utils_temp_7xma5yyz.xml'\nexit_status: 1\nduration: 0.029788970947265625\ninterrupted: False\npid: 30074\nencoding: 'UTF-8'\nstdout: '\n'\nstderr: "error: Failed to a... (49.09 s) TODO3 ===== for email/web report feature3.casexxx.log => job-2020-12-08T20.08-3bf7720/job.log (in result_root/) JOB LOG : /root/avocado/job-results/job-2020-12-08T20.08-3bf7720/job.log TODO4 ===== featureXXX.result.ok => featureXXX.nr_pass featureXXX.result.all => featureXXX.nr_total featureXXX.result.rate => featureXXX.pass_rate (perhaps not that useful, since the total pass_rate looks more useful, which can only be calculated by the framework across multi-jobs) RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 => stats => libvirt_usb_hotplug_device.nr_pass: 0 (if 0, may not print this line) libvirt_usb_hotplug_device.nr_fail: 1 (for account report) libvirt_usb_hotplug_device.nr_error: 0 libvirt_usb_hotplug_device.nr_warn: 0 libvirt_usb_hotplug_device.nr_total: 1 Thanks, Fengguang

1 0

[meeting] stats rules
by Lu Weitao 14 Feb '22

14 Feb '22

# type fail 0/1 counter N perf 自然数，float text [string]/string datetime? epoch? number? is array or scalar? is kpi? is negative corelation? perf split to throughput, latency, power? # per-script definition etc/failure ~/lkp-tests/etc/event-counter-prefixes ~/lkp-tests/etc/independent-counter-prefixes 1|index-perf-all.yaml 2|index-perf.yaml 3|perf-metrics-patterns 4|perf-metrics-prefixes 5|perf-metrics-threshold.yaml are all global what if a user to add one single test case and script? best to only define it in a self maintained dir with yaml description tests/unixbench + tests/unixbench.yaml os_depends: ubuntu: unixbench input.params: # will be indexed by ES nr_task: description: #type: int # for lint and catch early typo at submit time unit: default: test: input.env: # won't be indexed by ES output.stats: score match by name by regex by prefix type: record: by regex start_time .*\.message result_root/stats.json result_root/record.json # stats global stats pattern *.(log|message|element): type: string script stats pattern buildinfo\..*: type: hash example proc-vmstat pattern nr_.*: type perf pg.*: counter xxx yyy zzz: counter etc/event-counter-prefixes:proc-vmstat.nr_promoted etc/event-counter-prefixes:proc-vmstat.nr_promote_fail etc/event-counter-prefixes:proc-vmstat.nr_promote_ratelimit etc/event-counter-prefixes:proc-vmstat.nr_promote_isolate_fail etc/event-counter-prefixes:proc-vmstat.allocstall etc/event-counter-prefixes:proc-vmstat.compact_ etc/event-counter-prefixes:proc-vmstat.htlb_buddy_alloc_ etc/event-counter-prefixes:proc-vmstat.kswapd_ etc/event-counter-prefixes:proc-vmstat.nr_dirtied etc/event-counter-prefixes:proc-vmstat.nr_vmscan_immediate_reclaim etc/event-counter-prefixes:proc-vmstat.nr_written etc/event-counter-prefixes:proc-vmstat.numa_ etc/event-counter-prefixes:proc-vmstat.pageoutrun etc/event-counter-prefixes:proc-vmstat.pg etc/event-counter-prefixes:proc-vmstat.pswp etc/event-counter-prefixes:proc-vmstat.slabs_scanned etc/event-counter-prefixes:proc-vmstat.thp_ etc/event-counter-prefixes:proc-vmstat.unevictable_pgs_ etc/event-counter-prefixes:proc-vmstat.zone_reclaim_failed problem: this is not clean namespace split log.xxx: xxx: solution1: move log out of stats? job.stats.buildinfo.hash from $(stats stdout), $script.yaml job.record.buildinfo.env from $script.record.yaml also job.stats.xxx.message: string/[] stats only has number types? # stats namespace change implement

1 0

[todo] auto submit cci-makepkg for all tests
by Cao Xueliang 14 Feb '22

14 Feb '22

# auto submit cci-makepkg before run test suite example: - job.yaml --- '#! rpmbuild.yaml': suite: rpmbuild category: functional rpmbuild: repo_addr: https://vault.centos.org/6.10/os/Source/SPackages/ed-1.1-3.3.el6.src.rpm arch: aarch64 custom_repo_name: unstable/compatible/6 mount_repo_addr: https://api.compass-ci.openeuler.org:20018/rpm/testing/openeuler-20.03-LTS-… 1/compat-centos6/standard_\$basearch/\$basearch mount_repo_name: compatible-6 docker_image: openeuler:20.03-LTS-SP1 ``` - generate cci-makepkg.yaml # detect $LKP_SRC/pkg/rpmbuild/PKGBUILD first # detect /srv/initrd/pkg/container/openeuler/aarch64/20.03-LTS-SP1/rpmbuild/latest.cgz ``` suite: cci-makepkg category: functional cci-makepkg: benchmark: rpmbuild # from rpmbuild.yaml os: # from rpmbuild.yaml os_arch: # from rpmbuild.yaml os_mount: # from rpmbuild.yaml ``` - submit cci-makepkg.yaml & rpmbuild.yaml - if cci-makepkg job success and exists latest.cgz then submit rpmbuild.yaml - if cci-makepkg job not success then echo error & no need submit rpmbuild.yaml

1 0

Re: [PATCH v3 compass-ci] data-api: add es search api
by Wu Fengguang 11 Feb '22

11 Feb '22

On Wed, Feb 09, 2022 at 09:23:20PM +0800, weijihui wrote: >On Wed, Feb 09, 2022 at 01:21:14PM +0800, Wu Fengguang wrote: >>On Tue, Feb 08, 2022 at 03:10:57PM +0800, Wei Jihui wrote: >>>From: Wei Jihui <weijihuiall(a)163.com> >>> >>>1. /data_api/es/:index/es_search is es search by dsl >> >>是这样的吧： >> >> es_search => _search > >yes > >> >>>2. /data_api/es/:index/es_search_by_sql is es search by sql >> >>我看你后面用了 >> >> >+# curl -X POST localhost:10005/data_api/es/jobs/_search_by_sql >> >>然后调用的是 >> >> >+ @client.perform_request('GET', '_nlpcn/sql', {}, search_sql) >> >>但我们应该最终调用这个吧: >> >> https://github.com/opendistro-for-elasticsearch/sql >> >> POST https://<host>:<port>/_opendistro/_sql >> { >> "query": "SELECT * FROM my-index LIMIT 50" >> } >> >>相应的API是不是相应的写成 >> >> /data_api/_opendistro/_sql > >这个接口需要安装的插件没有我们用的es版本 >目前用的这个接口 >_nlpcn/sql 可以计划下升级到 _opendistro 和与之匹配的ES。 API层面，可以先用 _opendistro 免得折腾。 >> >>>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>>+# curl -X POST localhost:10005/data_api/es/jobs/_search -H 'Content-Type: application/json' -d '{ >>> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >>>-# "index": "jobs", >>>-# "my_token": "16132550-...", >>>-# "my_account": "auto-submit", >>>-# "query_type": "eql" # can be omitted in this scenario >>>+# "cci_authorization": { >> >>这个貌似不错： >> >> credentials > >cci_credentials? Yes. Thanks, Fengguang

1 0

Re: [PATCH todo 1/2] add todo for docker improvement
by Luan Shengde 11 Feb '22

11 Feb '22

pls ignore this mail Thanks, Luan Shengde On Fri, Feb 11, 2022 at 05:05:52PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > .../todo/20220216-5d:docker-improvement.md | 35 +++++++++++++++++++ > 1 file changed, 35 insertions(+) > create mode 100644 people/LuanShengde/todo/20220216-5d:docker-improvement.md > >diff --git a/people/LuanShengde/todo/20220216-5d:docker-improvement.md b/people/LuanShengde/todo/20220216-5d:docker-improvement.md >new file mode 100644 >index 00000000..416c749d >--- /dev/null >+++ b/people/LuanShengde/todo/20220216-5d:docker-improvement.md >@@ -0,0 +1,35 @@ >+# docker improvement >+ >+## handle download failure and check dir besides /lkp and /opt >+ >+original: >+ curl_cmd: %x(curl -sS --create-dirs -o #{path}/#{name} #{url} && gzip -dc #{path}/#{name} | cpio -idu -D #{path}) >+ download cgz file and unpack it in one line >+ >+improvement: >+ curl_cmd: split download cgz file, check dir and unpack the cgz file into standalone lines. >+ system('wget -t 3 -T 20 -O #{path}/#{name} #{url} 2> /dev/null') >+ use system to call the linux command, it will retusn true/false >+ - true: execute success >+ - false: execute in failure >+ raise error if download in failure >+ >+ %x(gzip -dc path/cgz_file | cpio -idt) >+ use %x(command) can get the output of the command, extract the file list of the cgz file >+ check files from the file list and check if the file/dir is besides the /lkp or /opt >+ >+ if no files/dirs besides /lkp or /opt >+ %x(gzip -dc path/cgz_file | cpio -idu -D path) >+ >+ if exist files/dirs besides /lkp or /opt >+ save the cgz to opt >+ unpack the cgz file in container >+ >+ --- >+ maybe can ignore check the cgz file if it containers files/dirs besides /lkp or /opt >+ just unpack all the cgz file in the container >+ --- >+ >+## /usr/local/bin/busybox should be ro >+ >+## cancel mount unnecessary dir >-- >2.23.0 >

1 0

Re: [PATCH v2 compass-ci] assign-account-tool: update email content
by Luan Shengde 11 Feb '22

11 Feb '22

On Thu, Feb 10, 2022 at 09:53:48PM +0800, Wei Jihui wrote: >On Thu, Feb 10, 2022 at 03:58:43PM +0800, Luan Shengde wrote: >>From: Luan Shengde <shdluan(a)163.com> >> >>for only login with public key is allow now, update the email content. >> >>Signed-off-by: Luan Shengde <shdluan(a)163.com> >>--- >> .../assign-account/build-send-account-email.rb | 13 ++++++++++--- >> container/assign-account/build-update-email.rb | 13 +++++++++++-- >> 2 files changed, 21 insertions(+), 5 deletions(-) >> >>diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb >>index 3c414041..f30de78b 100755 >>--- a/container/assign-account/build-send-account-email.rb >>+++ b/container/assign-account/build-send-account-email.rb >>@@ -16,10 +16,17 @@ def build_message(email, account_info) >> Login command: >> ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} >> >>- Account password: >>- #{account_info['my_password']} >>+ You can directly use the public key you offered to login the account-ver server. >> >>- You can use your pub_key to login if you have offered one. >>+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >>+ >>+ name: Zhang Dewan >>+ phone: 18969058870 >>+ email: zhangdewan1(a)compass-ci.org >>+ >>+ name: Luan Shengde >>+ phone: 15109218229 >>+ email: luanshengde(a)compass-ci.org >> >> Suggest: >> If you use the password to login, change it in time. >>diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb >>index 40d6b3b3..0a298ac8 100755 >>--- a/container/assign-account/build-update-email.rb >>+++ b/container/assign-account/build-update-email.rb >>@@ -18,8 +18,17 @@ def build_update_message(email, account_info, conf_info) >> Login command: >> ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} >> >>- Account password: >>- #{account_info['my_password']} >>+ You can directly use the public key you offered to loging the jumper server. > > loging --> login? > > Thanks, > Jihui OK, I will fix it Thanks, Luan Shengde > > >>+ >>+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >>+ >>+ name: Zhang Dewan >>+ phone: 18969058870 >>+ email: zhangdewan1(a)compass-ci.org >>+ >>+ name: Luan Shengde >>+ phone: 15109218229 >>+ email: luanshengde(a)compass-ci.org >> >> regards >> compass-ci >>-- >>2.23.0 >>

1 0

Re: [PATCH v2 compass-ci] assign-account-tool: update email content
by weijihui 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 03:58:43PM +0800, Luan Shengde wrote: >From: Luan Shengde <shdluan(a)163.com> > >for only login with public key is allow now, update the email content. > >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > .../assign-account/build-send-account-email.rb | 13 ++++++++++--- > container/assign-account/build-update-email.rb | 13 +++++++++++-- > 2 files changed, 21 insertions(+), 5 deletions(-) > >diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb >index 3c414041..f30de78b 100755 >--- a/container/assign-account/build-send-account-email.rb >+++ b/container/assign-account/build-send-account-email.rb >@@ -16,10 +16,17 @@ def build_message(email, account_info) > Login command: > ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > >- Account password: >- #{account_info['my_password']} >+ You can directly use the public key you offered to login the account-ver server. > >- You can use your pub_key to login if you have offered one. >+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >+ >+ name: Zhang Dewan >+ phone: 18969058870 >+ email: zhangdewan1(a)compass-ci.org >+ >+ name: Luan Shengde >+ phone: 15109218229 >+ email: luanshengde(a)compass-ci.org > > Suggest: > If you use the password to login, change it in time. >diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb >index 40d6b3b3..0a298ac8 100755 >--- a/container/assign-account/build-update-email.rb >+++ b/container/assign-account/build-update-email.rb >@@ -18,8 +18,17 @@ def build_update_message(email, account_info, conf_info) > Login command: > ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > >- Account password: >- #{account_info['my_password']} >+ You can directly use the public key you offered to loging the jumper server. loging --> login? Thanks, Jihui >+ >+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >+ >+ name: Zhang Dewan >+ phone: 18969058870 >+ email: zhangdewan1(a)compass-ci.org >+ >+ name: Luan Shengde >+ phone: 15109218229 >+ email: luanshengde(a)compass-ci.org > > regards > compass-ci >-- >2.23.0 >

1 0

Re: reproducible-build TODO list
by weijihui 10 Feb '22

10 Feb '22

On Wed, Feb 09, 2022 at 10:35:20AM +0800, Lu Weitao wrote: >On Tue, Feb 08, 2022 at 08:20:37PM +0800, Wei Jihui wrote: >>On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >>>## TODO list >>>- TODO1.1: script support set env for build env --- compass-ci jihui >> >>what should i do for this? >> > >user submit rpmbuild.yaml will with some build params like: >fs: >kernel: >reproducible_build_env: > hostname: > datetime: >rpmbuild: > new_param: > >we should make sure these params take effect in testbox no need reproducible_build_env, just write it like runtime. runtime: 1d hostname: cc datetime: bb Thanks, Jihui > > > >--------- >Thanks, >Lu Weitao > >- > >>Thanks, >>Jihui >> >> >>> >>>- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci >>> - based on debian buildinfo >>> - TODO1.3: 输出用户指定的息into buildinfo? >>> new lib/buildinfo.sh --- IT >>> func1() >>> func2() >>> func3() >>> save to $result_root/buildinfo.yaml >>> >>> modify rpmbuild script to call the above funcs. --- compass-ci wangyong >>> >>>- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao >>> >>>- TODO2: 比较构建结果是否相同 --- compass-ci weitao >>> compass ci >>> >>> reuse data API es-find >>> compare in client side >>> >>> 格式和进一步优化: IT >>> client: ask use python? >>> >>>- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT >>> >>>- TODO4 toolset --- IT/compass-ci >>> option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 >>> option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT >>> option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong >>> >>> >>> >>>--------- >>>Thanks, >>>Lu Weitao >>>

1 0

Re: [TODO] draft 2022 targets
by Bai Jing 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 04:14:55PM +0800, Wu Fengguang wrote: >One more: > >Setup and run syzkaller, keep trigger kernel bugs and reproduce program >accumulate and reuse the above reproduce programs as kernel test case > >Thanks, >Fengguang > >On Wed, Feb 09, 2022 at 05:19:14PM +0800, Wu Fengguang wrote: >>众智项目候选 >>界面 todo owner >>- web interface luweitao >>- maintain dashboard zhangdewan >>加包 >>- OS package name mapping wuzhende >>- auto convert 3w archlinux packages wangyong >>- auto create 3w C/C++ spec liping >>- test 10w pypi upstream packages liyuanchao >>- rpm spec customize options from gentoo and ./configure output parsing >>nix >>- nix 1w upstream project tests >>- openeuler 3k overlay on nixpkgs >>- openEuler+Nix+musl docker images for popular apps, replacing alpine >>改进? >>- k8s integration luanshengde >>- selftest liuyinsi >>- bisect caoxueliang >> >>OS大赛项目候选 >>- YAML+python overlays merge tool >> >>Thanks, >>Fengguang >>

1 0

腾讯企业邮箱自动转发验证邮件
by luanshengde 10 Feb '22

10 Feb '22

亲爱的朋友：您好，腾讯企业邮箱用户 luanshengde(a)compass-ci.org 在邮箱中设置了将邮件转发至您的邮箱中。为避免不友善的转发给您带来骚扰，该设置需要您允许才能生效。如果您允许该转发设置，请点击以下链接：接受该转发请求   如果您点击上述链接无效，您可能处于纯文本模式下。请复制以下地址至浏览器地址栏进行访问： http://exmail.qq.com/cgi-bin/attrset?fun=autoreply&t=verifyfw_result&alias=…     腾讯企业邮箱腾讯企业邮箱门户：http://exmail.qq.com 企业邮箱帮助中心：http://service.exmail.qq.com

1 0

test mail for forward
by Cao Xueliang 10 Feb '22

10 Feb '22

test mail for forward

1 0

Re: [TODO] draft 2022 targets
by Bai Jing 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 04:14:55PM +0800, Wu Fengguang wrote: >One more: > >Setup and run syzkaller, keep trigger kernel bugs and reproduce program >accumulate and reuse the above reproduce programs as kernel test case > >Thanks, >Fengguang > >On Wed, Feb 09, 2022 at 05:19:14PM +0800, Wu Fengguang wrote: >>众智项目候选 >>界面 todo owner >>- web interface luweitao >>- maintain dashboard zhangdewan >>加包 >>- OS package name mapping wuzhende >>- auto convert 3w archlinux packages wangyong >>- auto create 3w C/C++ spec liping >>- test 10w pypi upstream packages liyuanchao >>- rpm spec customize options from gentoo and ./configure output parsing >>nix >>- nix 1w upstream project tests >>- openeuler 3k overlay on nixpkgs >>- openEuler+Nix+musl docker images for popular apps, replacing alpine >>改进? >>- k8s integration luanshengde >>- selftest liuyinsi >>- bisect caoxueliang >> >>OS大赛项目候选 >>- YAML+python overlays merge tool >> >>Thanks, >>Fengguang >>

1 0

Re: extract buildinfo
by Lu Weitao 10 Feb '22

10 Feb '22

please help review this design --------- Thanks, Lu Weitao On Wed, Feb 09, 2022 at 05:20:12PM +0800, Lu Weitao wrote: ># extract buildinfo and save it to ES > todo from reproducible-build flow > > ``` > after build rpm, and upload buildinfo.yaml, rpmbuild to server > | > v > $result_root/rpmbuild.yaml ---> rpmbuild.json ---> merge to stats.json > $result_root/rpmbuild_info.yaml ---> rpmbuild_info.json ---> merge to stats.json > | > v > save stats.json to ES db # 为compare， diifoscope 提供数据源 > eg: > { > "id": xxx, > "os": "openeuler", > "arch": "aarch64" > "stats": { > "rpmbuild.hash_value": xxx, > ``` >## TODO1: improve compass-ci/sbin/result2stats, extract-stats service call it > support extract buildinfo > usage: result2stats $result_root > ``` > ... > $result_root/rpmbuild ---> rpmbuild.json ---> merge to stats.json > $result_root/buildinfo.yaml ---> buildinfo.json ---> merge to stats.json > ... > ``` >### input: result_root/buildinfo(buildinfo.yaml) >- e.g. samba_4.9.5+dfsg-5+deb10u2_arm64.buildinfo >``` >Format: 1.0 >Source: samba >Binary: samba samba-libs samba-common samba-common-bin smbclient samba-testsuite registry-tools samba-dev python-samba samba-dsdb-modules samba-vfs-modules libsmbclient libsmbclient-dev winbind libpam-winbind libnss-winbind libwbclient0 libwbclient-dev ctdb >Architecture: all arm64 source >Version: 2:4.9.5+dfsg-5+deb10u2 >Checksums-Md5: > 79ffac5466e2b36633e0f429644407d0 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > d523774b75cc4d31dce4696d41d68b8f 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Checksums-Sha1: > b8c10499c1b6b7c5111c7f2d8d617ed8a2ff938e 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > 1569bfdb4399f92852863f67153cce046a0cbb38 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Checksums-Sha256: > c9094fcc63930f665707c596134ea64e44d433d3882b3de544c2a83f574aafb9 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > 3304c2fa4f7204682747bed70f7ce61b134b21f063f9ebab0aeb2e3af645b744 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Build-Origin: Debian >Build-Architecture: arm64 >Build-Date: Sun, 30 Jan 2022 09:27:50 +0000 >Installed-Build-Depends: > adduser (= 3.118), > zlib1g-dev (= 1:1.2.11.dfsg-1) > ... >Environment: > DEB_BUILD_OPTIONS="parallel=128" > SOURCE_DATE_EPOCH="1638005690" >``` >### output result_root/buildinfo.json >``` >{ > "Hash": "SHA512", > "Format": 1.0, > "Build-Architecture": "i386", > "Source": "fweb (1.62-12)", > "Binary": "fweb fweb-doc", > "Architecture": "all i386", > "Version": "1.62-12+b2", > "Binary-Only-Changes": > "fweb (1.62-12+b2) sid; urgency=low, binary-only=yes > . > * Binary-only non-maintainer upload for amd64; no source changes. > * Rebuild for multiarch sync > . > -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms(a)buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000", > ... >} >``` > > >## TODO2: save buildinfo to job(ES) > extract-service load $result_root/buildinfo.json > >- option1: merge job['buildinfo'](job['record']['buildinfo']) > meeting record: > job.record.build.xxx: v => possible solution option1 > 通用性好（job['record']['buildinfo'][xxx]） > 结构较复杂 > > job.info.build.xxx: v > 效果同上 > value type: > - output binary hash > - info (xxx.rpm) > > job['buildinfo'] > 结构简单，但只适用与构建任务，目前未发现其它类型的job有这种需求，如需使用buildinfo进行搜索，那这种方式更简单 > > **job like:** > ``` > { > "id": xxx, > "os": "openeuler", > ... > "buildinfo": { > "Hash": "SHA512", > "Format": 1.0, > ... > } > } > ``` > **advantage:** > 1. reuse job['buildinfo'] to submit again # we can also merge it to job.yaml > 2. 3rd tool such as: `diffoscope` get job['buildinfo'] > **disadvantage:** > 1. compare need get metric from buildinfo in addition > >- option2: merge buildinfo to job['stats'], job like: > ``` > { > "id": xxx, > "os": "openeuler", > "stats": { > "buildinfo.Hash": "SHA512", > "buildinfo.Format": 1.0, > ... > } > } > ``` > advantage / disadvantage: opposite of option1 > > job.stats.rpmbuild.xxx > job['stats']['rpmbuild.xxx'] > compare 不好区分buildinfo 和 rpmbuild > 需要变动lkp-tests/stats/rpmbuild，且buildinfo > result_root/rpmbuild > > job.stats.buildinfo.xxx => possible solution option2 > job['stats']['buildinfo.xxx'] > extract-stats 已经支持，无需改动代码 > 仅仅将buildinfo信息输出未 result_root/buildinfo.yaml > > compare 性能结果时忽略 buildinfo （可以新加perf-metric-ignore规则） > > job.stats.message.xxx / job.stats.xxx.message > 目前.message, .log 已经是perf-metric-ignore规则，可以考虑将buildinfo也加进去 > > > job.stats.info.build.buildinfo.xhhhhhxx => possible solution > 通用性更好，相当于对metric有个全面的分类 > 结构较多 > > > job.buildinfo.xxx（in option1） > >- option3: option1 + option2 > job like: > ``` > { > "id": xxx, > "os": "openeuler", > ... > "buildinfo": { > "Hash": "SHA512", > "Format": 1.0, > ... > }, > "stats": { > "buildinfo.Hash": "SHA512", > "buildinfo.Format": 1.0, > ... > } > } > > ``` > **advantage: opton1 + option2 ** > **disadvantage:** > 1. job bigger, --> save DB > 2. extract-service does one more thing >_______________________________________________ >Compass-ci mailing list -- compass-ci(a)openeuler.org >To unsubscribe send an email to compass-ci-leave(a)openeuler.org

1 0

Re: [PATCH v7 compass-ci] for test
by Luan Shengde 10 Feb '22

10 Feb '22

Current branch master is up to date. TO team leader: caoxueliang(a)compass-ci.org weijihui(a)compass-ci.org wufengguang(a)compass-ci.org --to=luanshengde(a)compass-ci.org TO file committer: luanshengde(a)compass-ci.org CC file author: luanshengde(a)compass-ci.org git send-email -q --no-thread --confirm=never --cc=all --cc=compass-ci(a)openeuler.org --to=caoxueliang(a)compass-ci.org --to=weijihui(a)compass-ci.org --to=wufengguang(a)compass-ci.org --to=luanshengde(a)compass-ci.org - -cc=luanshengde(a)compass-ci.org 0001-for-test-v8.patch On Thu, Feb 10, 2022 at 03:39:52PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <luanshengde(a)compass-ci.org> >--- > aaa | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 aaa > >diff --git a/aaa b/aaa >new file mode 100644 >index 00000000..4aed24e8 >--- /dev/null >+++ b/aaa >@@ -0,0 +1 @@ >+test msg >-- >2.23.0 >

1 0

Re: [PATCH v7 compass-ci] for test
by Luan Shengde 10 Feb '22

10 Feb '22

ignore this mail On Thu, Feb 10, 2022 at 03:39:52PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <luanshengde(a)compass-ci.org> >--- > aaa | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 aaa > >diff --git a/aaa b/aaa >new file mode 100644 >index 00000000..4aed24e8 >--- /dev/null >+++ b/aaa >@@ -0,0 +1 @@ >+test msg >-- >2.23.0 >

1 0

Re: [meeting] install-depends summary & TODO list
by Wang Yong 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 11:22:30AM +0800, Wang Yong wrote: ># direct install depends > >- submit job.yaml >+ add a field to job.yaml from distro/depends/$job >fio: tests/ >fs: setup/ > install_depends: # add to setup/ > fs: pkg1 pkg2 > host-info: pkg3 > $script: pkg1 > >+ change pkgs' name from distro/adaptation/$distro >+ export $install_depends_packages to install_depends > >- setup/install_depends: > install packages due to $packages from YAML > >leftover: > - CCI will check, download, install initrd_deps always > > delete all deps will fix it > - share_debian_package ( run-ipconfig ) > > not work anymore > commit 8a8a57d7a166687bad1cd11e880a36c80e2c403b > > > adapt from debian to $os > ~/lkp-tests/distro/depends/$script (debian) > => > ~/lkp-tests/distro/depends/debian/$script > ~/lkp-tests/distro/adaptation/$os > > bottom line: on error, user is able to fix it by specifying exact dependencies in a depends/ file > >x * y >x y1 y2 y3 y4 > 1 2 11 12 > > find package in os similarity order > debian-11/$script > debian/$script > deb os (if os is ubuntu) > show warning on submit > > ubuntu user: lib-dev > > ~/lkp-tests/distro/depends/$os/$script (user development os) > job: $os no problem > > job: $other_os auto convert in > - submit > - scheduler > call service API: convert package name from one os to another os > >TODO: > - add filed into job.yaml with all setup/tests script --Zhende > '#! /home/wangyong/lkp-tests/jobs/host-info.yaml': > suite: host-info > category: functional > is_send_mail: false > host-info: > os: openeuler > os_arch: aarch64 > os_version: "20.03" > install_depends_pakeages: > fs: pkg1 pkg2 > host-info: pkg3 pkg4 > - convert debian/$script packages into target $os packages --Zhende > adapt from debian to $os > ~/lkp-tests/distro/depends/$script (debian) > => > ~/lkp-tests/distro/depends/debian/$script > ~/lkp-tests/distro/adaptation/$os > - copy /depends/$script to /depends/debian/$script --Wangyong > - move setup/install_depends from run_job to lkp-tests frame --Wangyong > - to prevent install depends after setup initialization > - similarity tree to list most similar rpm/deb to install --Zhende >_______________________________________________ >Compass-ci mailing list -- compass-ci(a)openeuler.org >To unsubscribe send an email to compass-ci-leave(a)openeuler.org

1 0

[meeting] install-depends summary & TODO list
by Wang Yong 10 Feb '22

10 Feb '22

# direct install depends - submit job.yaml + add a field to job.yaml from distro/depends/$job fio: tests/ fs: setup/ install_depends: # add to setup/ fs: pkg1 pkg2 host-info: pkg3 $script: pkg1 + change pkgs' name from distro/adaptation/$distro + export $install_depends_packages to install_depends - setup/install_depends: install packages due to $packages from YAML leftover: - CCI will check, download, install initrd_deps always > delete all deps will fix it - share_debian_package ( run-ipconfig ) > not work anymore commit 8a8a57d7a166687bad1cd11e880a36c80e2c403b adapt from debian to $os ~/lkp-tests/distro/depends/$script (debian) => ~/lkp-tests/distro/depends/debian/$script ~/lkp-tests/distro/adaptation/$os bottom line: on error, user is able to fix it by specifying exact dependencies in a depends/ file x * y x y1 y2 y3 y4 1 2 11 12 find package in os similarity order debian-11/$script debian/$script deb os (if os is ubuntu) show warning on submit ubuntu user: lib-dev ~/lkp-tests/distro/depends/$os/$script (user development os) job: $os no problem job: $other_os auto convert in - submit - scheduler call service API: convert package name from one os to another os TODO: - add filed into job.yaml with all setup/tests script --Zhende '#! /home/wangyong/lkp-tests/jobs/host-info.yaml': suite: host-info category: functional is_send_mail: false host-info: os: openeuler os_arch: aarch64 os_version: "20.03" install_depends_pakeages: fs: pkg1 pkg2 host-info: pkg3 pkg4 - convert debian/$script packages into target $os packages --Zhende adapt from debian to $os ~/lkp-tests/distro/depends/$script (debian) => ~/lkp-tests/distro/depends/debian/$script ~/lkp-tests/distro/adaptation/$os - copy /depends/$script to /depends/debian/$script --Wangyong - move setup/install_depends from run_job to lkp-tests frame --Wangyong - to prevent install depends after setup initialization - similarity tree to list most similar rpm/deb to install --Zhende

1 0

Re: [PATCH v3 compass-ci] data-api: add es search api
by weijihui 09 Feb '22

09 Feb '22

On Wed, Feb 09, 2022 at 01:21:14PM +0800, Wu Fengguang wrote: >On Tue, Feb 08, 2022 at 03:10:57PM +0800, Wei Jihui wrote: >>From: Wei Jihui <weijihuiall(a)163.com> >> >>1. /data_api/es/:index/es_search is es search by dsl > >是这样的吧： > > es_search => _search yes > >>2. /data_api/es/:index/es_search_by_sql is es search by sql > >我看你后面用了 > > >+# curl -X POST localhost:10005/data_api/es/jobs/_search_by_sql > >然后调用的是 > > >+ @client.perform_request('GET', '_nlpcn/sql', {}, search_sql) > >但我们应该最终调用这个吧: > > https://github.com/opendistro-for-elasticsearch/sql > > POST https://<host>:<port>/_opendistro/_sql > { > "query": "SELECT * FROM my-index LIMIT 50" > } > >相应的API是不是相应的写成 > > /data_api/_opendistro/_sql 这个接口需要安装的插件没有我们用的es版本目前用的这个接口 _nlpcn/sql > >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es/jobs/_search -H 'Content-Type: application/json' -d '{ >> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >>-# "index": "jobs", >>-# "my_token": "16132550-...", >>-# "my_account": "auto-submit", >>-# "query_type": "eql" # can be omitted in this scenario >>+# "cci_authorization": { > >这个貌似不错： > > credentials cci_credentials? Thanks, Jihui > > 1 ~ (for/as sth); ~ (to do sth) qualities, achievements, etc that make one suitable; qualifications （学历、资历等的）资格: She has the perfect credentials for the job. 她做这工作完全够格. > 2 documents showing that a person is what he claims to be, is trustworthy, etc （某人合乎条件、可以信任等的）证明书, 证件: I examined his credentials. 我查验了他的证件. > >相比而言，这更像是一个结果： > > authorization > n. 授权, 许可 > >Thanks, >Fengguang >

1 0

Re: [TODO] provide LKP dependencies in container with busybox+curl
by Luan Shengde 09 Feb '22

09 Feb '22

- use debian OS as host OS - make sure host installed busybox-static package and downloaded static-curl - mount busybox/curl into docker - create busybox symlinks in docker mkdir /opt/busybox/bin/ cd /opt/busybox/bin/ for c in wget cpio do ln -s /path/to/busybox $c done PATH=$PATH:/opt/busybox/bin/ by fengguang thanks - download .cgz depends inside docker in LKP framework - extract .cgz On Mon, Feb 07, 2022 at 02:26:54PM +0800, Wu Fengguang wrote: >Currently /c/compass-ci/providers/docker/run.sh has extra dependencies >that should be simplified. > >TODO1: install busybox in the host (I see no script to make sure it's >installed in the host). If the host OS does not has the busybox >package, it's fine to download pre-built busybox-static binary > >TODO2: make sure busybox symlinks are created in a dir like >/opt/busybox/bin/ inside the container, and _append_ it to PATH. > >TODO3: install curl-static in the host and mount into container >/opt/curl/bin/ > > https://github.com/moparisthebest/static-curl > > Grab the latest release from one of these links, by CPU architecture: > curl-amd64 > curl-i386 > curl-aarch64 > curl-armv7 > curl-ppc64le > >busybox+curl should provide all the *required* dependencies for >running LKP framework. > >Thanks, >Fengguang > >On Tue, Jan 04, 2022 at 10:07:08AM +0800, Wu Fengguang wrote: >>Let's cut down dependencies in /c/lkp-tests/distro/depends/lkp >> >>For the depends added by >> >> commit 54e515a10156080bfabc8d880b3f704b7c29a702 >> Author: Wei Jihui <weijihuiall(a)163.com> >> AuthorDate: Mon Nov 8 09:59:15 2021 +0800 >> Commit: Wei Jihui <weijihuiall(a)163.com> >> CommitDate: Thu Nov 11 14:25:03 2021 +0800 >> >> distro/depends/lkp: move cci-depends'depends to lkp's depends >> >>we can use either >>- busybox ftpput >>- rsync >>- lftp >>for uploading files. Those packages are rarely depended by normal >>packages. In comparison, curl is depended by hundreds of other >>packages in Debian. >> >>If use busybox, lkp can remove lots of dependencies. >> >>However for remote testbox, its 'busybox ftpput' command may not work well >>- need PASSIVE mode in order to go through firewall >>- not secure over Internet >>- some firewall may bann insecure service/ports >>- not good to block testbox in slow/unreliable Internet connection >> >>The same for rsync upload, if ssh is not used together. >> >>But shall we upload all remote testbox results over Internet? >>I guess not -- it won't scale. >> >>Here is a flexible upload model >> >>testbox >>-> on job finish, tar/cpio all results into a single file >> (to avoid complex mirror function for ftpput) >>-> upload via any of ftp/rsync/https to a dedicated and temp server storage space >> (eg. /srv/result/upload/$jobid/ with md5sum verification) >> (proftpd: configure as upload only dir, disable read and list-dir) >>-> tell scheduler that the job is complete >>-> run a service to unpack a job's results into final result_root >> >>remote testbox >>-> results uploaded to a local server storage >>-> local storage server runs a service to send results to central lab storage >> when asked by end user >> >>Thanks, >>Fengguang

1 0

extract buildinfo
by Lu Weitao 09 Feb '22

09 Feb '22

# extract buildinfo and save it to ES todo from reproducible-build flow ``` after build rpm, and upload buildinfo.yaml, rpmbuild to server | v $result_root/rpmbuild.yaml ---> rpmbuild.json ---> merge to stats.json $result_root/rpmbuild_info.yaml ---> rpmbuild_info.json ---> merge to stats.json | v save stats.json to ES db # 为compare， diifoscope 提供数据源 eg: { "id": xxx, "os": "openeuler", "arch": "aarch64" "stats": { "rpmbuild.hash_value": xxx, ``` ## TODO1: improve compass-ci/sbin/result2stats, extract-stats service call it support extract buildinfo usage: result2stats $result_root ``` ... $result_root/rpmbuild ---> rpmbuild.json ---> merge to stats.json $result_root/buildinfo.yaml ---> buildinfo.json ---> merge to stats.json ... ``` ### input: result_root/buildinfo(buildinfo.yaml) - e.g. samba_4.9.5+dfsg-5+deb10u2_arm64.buildinfo ``` Format: 1.0 Source: samba Binary: samba samba-libs samba-common samba-common-bin smbclient samba-testsuite registry-tools samba-dev python-samba samba-dsdb-modules samba-vfs-modules libsmbclient libsmbclient-dev winbind libpam-winbind libnss-winbind libwbclient0 libwbclient-dev ctdb Architecture: all arm64 source Version: 2:4.9.5+dfsg-5+deb10u2 Checksums-Md5: 79ffac5466e2b36633e0f429644407d0 3211 samba_4.9.5+dfsg-5+deb10u2.dsc d523774b75cc4d31dce4696d41d68b8f 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Checksums-Sha1: b8c10499c1b6b7c5111c7f2d8d617ed8a2ff938e 3211 samba_4.9.5+dfsg-5+deb10u2.dsc 1569bfdb4399f92852863f67153cce046a0cbb38 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Checksums-Sha256: c9094fcc63930f665707c596134ea64e44d433d3882b3de544c2a83f574aafb9 3211 samba_4.9.5+dfsg-5+deb10u2.dsc 3304c2fa4f7204682747bed70f7ce61b134b21f063f9ebab0aeb2e3af645b744 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Build-Origin: Debian Build-Architecture: arm64 Build-Date: Sun, 30 Jan 2022 09:27:50 +0000 Installed-Build-Depends: adduser (= 3.118), zlib1g-dev (= 1:1.2.11.dfsg-1) ... Environment: DEB_BUILD_OPTIONS="parallel=128" SOURCE_DATE_EPOCH="1638005690" ``` ### output result_root/buildinfo.json ``` { "Hash": "SHA512", "Format": 1.0, "Build-Architecture": "i386", "Source": "fweb (1.62-12)", "Binary": "fweb fweb-doc", "Architecture": "all i386", "Version": "1.62-12+b2", "Binary-Only-Changes": "fweb (1.62-12+b2) sid; urgency=low, binary-only=yes . * Binary-only non-maintainer upload for amd64; no source changes. * Rebuild for multiarch sync . -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms(a)buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000", ... } ``` ## TODO2: save buildinfo to job(ES) extract-service load $result_root/buildinfo.json - option1: merge job['buildinfo'](job['record']['buildinfo']) meeting record: job.record.build.xxx: v => possible solution option1 通用性好（job['record']['buildinfo'][xxx]）结构较复杂 job.info.build.xxx: v 效果同上 value type: - output binary hash - info (xxx.rpm) job['buildinfo'] 结构简单，但只适用与构建任务，目前未发现其它类型的job有这种需求，如需使用buildinfo进行搜索，那这种方式更简单 **job like:** ``` { "id": xxx, "os": "openeuler", ... "buildinfo": { "Hash": "SHA512", "Format": 1.0, ... } } ``` **advantage:** 1. reuse job['buildinfo'] to submit again # we can also merge it to job.yaml 2. 3rd tool such as: `diffoscope` get job['buildinfo'] **disadvantage:** 1. compare need get metric from buildinfo in addition - option2: merge buildinfo to job['stats'], job like: ``` { "id": xxx, "os": "openeuler", "stats": { "buildinfo.Hash": "SHA512", "buildinfo.Format": 1.0, ... } } ``` advantage / disadvantage: opposite of option1 job.stats.rpmbuild.xxx job['stats']['rpmbuild.xxx'] compare 不好区分buildinfo 和 rpmbuild 需要变动lkp-tests/stats/rpmbuild，且buildinfo > result_root/rpmbuild job.stats.buildinfo.xxx => possible solution option2 job['stats']['buildinfo.xxx'] extract-stats 已经支持，无需改动代码仅仅将buildinfo信息输出未 result_root/buildinfo.yaml compare 性能结果时忽略 buildinfo （可以新加perf-metric-ignore规则） job.stats.message.xxx / job.stats.xxx.message 目前.message, .log 已经是perf-metric-ignore规则，可以考虑将buildinfo也加进去 job.stats.info.build.buildinfo.xhhhhhxx => possible solution 通用性更好，相当于对metric有个全面的分类结构较多 job.buildinfo.xxx（in option1） - option3: option1 + option2 job like: ``` { "id": xxx, "os": "openeuler", ... "buildinfo": { "Hash": "SHA512", "Format": 1.0, ... }, "stats": { "buildinfo.Hash": "SHA512", "buildinfo.Format": 1.0, ... } } ``` **advantage: opton1 + option2 ** **disadvantage:** 1. job bigger, --> save DB 2. extract-service does one more thing

1 0

Re: reproducible-build TODO list
by Lu Weitao 09 Feb '22

09 Feb '22

On Tue, Feb 08, 2022 at 08:20:37PM +0800, Wei Jihui wrote: >On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >>## TODO list >>- TODO1.1: script support set env for build env --- compass-ci jihui > >what should i do for this? > user submit rpmbuild.yaml will with some build params like: fs: kernel: reproducible_build_env: hostname: datetime: rpmbuild: new_param: we should make sure these params take effect in testbox --------- Thanks, Lu Weitao - >Thanks, >Jihui > > >> >>- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci >> - based on debian buildinfo >> - TODO1.3: 输出用户指定的息into buildinfo? >> new lib/buildinfo.sh --- IT >> func1() >> func2() >> func3() >> save to $result_root/buildinfo.yaml >> >> modify rpmbuild script to call the above funcs. --- compass-ci wangyong >> >>- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao >> >>- TODO2: 比较构建结果是否相同 --- compass-ci weitao >> compass ci >> >> reuse data API es-find >> compare in client side >> >> 格式和进一步优化: IT >> client: ask use python? >> >>- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT >> >>- TODO4 toolset --- IT/compass-ci >> option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 >> option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT >> option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong >> >> >> >>--------- >>Thanks, >>Lu Weitao >>

1 0

Re: [PATCH lkp-tests 3/3] lkp-tests/*: add new flow for install lkp depends
by Wang Yong 09 Feb '22

09 Feb '22

On Tue, Feb 08, 2022 at 11:28:38AM +0800, Wei Jihui wrote: >On Sun, Jan 30, 2022 at 08:58:23AM +0800, Wang Yong wrote: >>[why] >>install lkp depends before test suite to satisfy lkp-tests >> >>Signed-off-by: Wang Yong <wangyong187(a)huawei.com> >>--- >> lib/job2sh.rb | 2 ++ >> setup/install_depends | 49 +++++++++++++++++++++++++++++++++++-------- >> 2 files changed, 42 insertions(+), 9 deletions(-) >> >>diff --git a/lib/job2sh.rb b/lib/job2sh.rb >>index 4e8bf97fb..32caf2ade 100755 >>--- a/lib/job2sh.rb >>+++ b/lib/job2sh.rb >>@@ -316,6 +316,7 @@ class Job2sh < Job >> @monitors = available_programs(:monitors) >> @setups = available_programs(:setup) >> @programs = available_programs(:workload_elements) >>+ @lkp = {"install_depends"=>{}} > >how about use '@install_depends'? > ok Thanks, Yong

1 0

Re: reproducible-build TODO list
by weijihui 08 Feb '22

08 Feb '22

On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >## TODO list >- TODO1.1: script support set env for build env --- compass-ci jihui what should i do for this? Thanks, Jihui > >- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci > - based on debian buildinfo > - TODO1.3: 输出用户指定的息into buildinfo? > new lib/buildinfo.sh --- IT > func1() > func2() > func3() > save to $result_root/buildinfo.yaml > > modify rpmbuild script to call the above funcs. --- compass-ci wangyong > >- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao > >- TODO2: 比较构建结果是否相同 --- compass-ci weitao > compass ci > > reuse data API es-find > compare in client side > > 格式和进一步优化: IT > client: ask use python? > >- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT > >- TODO4 toolset --- IT/compass-ci > option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 > option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT > option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong > > > >--------- >Thanks, >Lu Weitao >

1 0

Re: [PATCH compass-ci] container/data-api/data-api: fix incorrect comments
by Lu Weitao 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 09:39:14AM +0800, Wei Jihui wrote: >On Mon, Feb 07, 2022 at 11:22:13AM +0800, Lu Weitao wrote: >>Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >>--- >> container/data-api/data-api | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >>diff --git a/container/data-api/data-api b/container/data-api/data-api >>index 2bd6e527..c1c8af8f 100755 >>--- a/container/data-api/data-api >>+++ b/container/data-api/data-api >>@@ -17,7 +17,7 @@ set :environment, :production >> >> # POST >> # eg1: >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es_find -H 'Content-Type: application/json' -d '{ > > >we will define new api name for data_api. > ok --------- Thanks, Lu Weitao >Thanks, >Jihui > > >> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >> # "index": "jobs", >> # "my_token": "16132550-...", >>@@ -25,7 +25,7 @@ set :environment, :production >> # "query_type": "eql" # can be omitted in this scenario >> # }' >> # eg2: >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es_find -H 'Content-Type: application/json' -d '{ >> # "query": "SELECT * FROM jobs WHERE ...", >> # "index": "jobs", >> # "my_token": "16132550-...", >>-- >>2.23.0 >>

1 0

Re: [PATCH v2 compass-ci] data-api: add es search api
by Wei Jihui 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 10:28:21AM +0800, Wu Fengguang wrote: >>+ def self.search(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] || {'query' => {}} >>+ >>+ return "index is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) > >这里的return会在上一级被作为HTTP 200 >但看起来是一个错误情况。是的，抛出错误更好一些。 Thanks, Jihui > >>+ def self.search_by_sql(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] >>+ >>+ return "query table is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) > >同上。 > >Thanks, >Fengguang >

1 0

Re: [PATCH v2 compass-ci] data-api: add es search api
by Wei Jihui 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 10:24:24AM +0800, Wu Fengguang wrote: >>+module EsDataApi >>+ ES_ACCOUNTS = ESClient.new(index: 'accounts') >>+ OPEN_INDEX = Set.new(['jobs']) >>+ REQUIRED_TOKEN_INDEX = Set.new(['jobs']) >>+ ES_QUERY_KEYWORD = Set.new(['term', 'match']) >>+ >>+ > >2 empty lines can be merged. ok. > >>+ def self.search(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] || {'query' => {}} >>+ >>+ return "index is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) >>+ >>+ if REQUIRED_TOKEN_INDEX.include?(index) >>+ cci_authorization = request_body['cci_authorization'] || {} >>+ my_account = cci_authorization['my_account'] >>+ my_token = cci_authorization['my_token'] >>+ raise "user authentication failed" unless verify_user(my_account, my_token) > >加上my_account，把错误信息变得友好些。 ok. user authentication failed，please check my_account and my_token. > >>+ > >函数体有点大。功能型代码块，适合独立拆分出去。 ok. >>+ def self.verify_user(my_account, my_token) >>+ raise "missed my_account" unless my_account >>+ raise "missed my_token" unless my_token >>+ >>+ query = {} >>+ query['my_account'] = my_account if my_account >>+ query['my_token'] = my_token if my_token > >这里的两个if前面判断过了，可省略 > ok Thanks, Jihui >>+ result = ES_ACCOUNTS.multi_field_query(query)['hits']['hits'] >>+ return nil if result.size == 0 >>+ >>+ result[0]['_source']['my_token'] == my_token >>+ end >>+end >>-- >>2.23.0 >>

1 0

Re: [PATCH compass-ci] src/lib/web_backend.rb: set page_num and page_size to a default value.
by Wei Jihui 08 Feb '22

08 Feb '22

On Mon, Feb 07, 2022 at 04:02:52PM +0800, Li Ping wrote: >[why] >if no parameter is transferred, the default value is not set, As a >result, data cannot be obtained. > 我已经撤销了这个patch,合入最新的patch。 Thanks, Jihui >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > src/lib/web_backend.rb | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > >diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb >index f47950e6..571d7c64 100644 >--- a/src/lib/web_backend.rb >+++ b/src/lib/web_backend.rb >@@ -1026,8 +1026,10 @@ end > def get_srpm_info(params) > begin > my_data = MyData.new >+ page_size = params['page_size'] || 10 >+ page_num = params['page_num'] || 0 > info=[] >- srpm_infos = my_data.get_srpm_info(size: params['page_size'].to_i, from: params['page_num'].to_i) >+ srpm_infos = my_data.get_srpm_info(size: page_size, from: page_num) > srpm_infos['hits']['hits'].each do |source| > info << source['_source'] > end >-- >2.23.0 >

1 0

Re: [PATCH lkp-tests 3/3] lkp-tests/*: add new flow for install lkp depends
by Wei Jihui 08 Feb '22

08 Feb '22

On Sun, Jan 30, 2022 at 08:58:23AM +0800, Wang Yong wrote: >[why] >install lkp depends before test suite to satisfy lkp-tests > >Signed-off-by: Wang Yong <wangyong187(a)huawei.com> >--- > lib/job2sh.rb | 2 ++ > setup/install_depends | 49 +++++++++++++++++++++++++++++++++++-------- > 2 files changed, 42 insertions(+), 9 deletions(-) > >diff --git a/lib/job2sh.rb b/lib/job2sh.rb >index 4e8bf97fb..32caf2ade 100755 >--- a/lib/job2sh.rb >+++ b/lib/job2sh.rb >@@ -316,6 +316,7 @@ class Job2sh < Job > @monitors = available_programs(:monitors) > @setups = available_programs(:setup) > @programs = available_programs(:workload_elements) >+ @lkp = {"install_depends"=>{}} how about use '@install_depends'? Thanks, Jihui > > unless job > job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job >@@ -336,6 +337,7 @@ class Job2sh < Job > out_line > out_line "\texport_top_env" > out_line >+ parse_hash [], @lkp > parse_hash [], job > out_line "}\n\n" > @script_lines >diff --git a/setup/install_depends b/setup/install_depends >index b5390cad1..8a6da808e 100755 >--- a/setup/install_depends >+++ b/setup/install_depends >@@ -1,6 +1,7 @@ > #!/bin/sh > > . $LKP_SRC/lib/detect-system.sh >+. $LKP_SRC/lib/install.sh > > detect_system > >@@ -16,8 +17,6 @@ set_ubuntu_debian() > -e "s|http://archive.ubuntu.com|${mirror_addr}|g" \ > -i.bak \ > /etc/apt/sources.list >- >- apt-get install -yqm "${packages}" > } > > set_suse() >@@ -25,8 +24,6 @@ set_suse() > sed -e "s|^baseurl=http://download.opensuse.org|baseurl=${mirror_addr}/opensuse|g" \ > -i.bak \ > /etc/zypp/repos.d/repo-*.repo >- >- zypper -q install -y "${packages}" > } > > set_archlinux() >@@ -36,8 +33,6 @@ set_archlinux() > else > sed -i "1s;^;Server\\ =\\ ${mirror_addr}/archlinux/\$arch/\$repo\\n;" /etc/pacman.d/mirrorlist > fi >- >- pacman -Syq --noconfirm --needed "${packages}" > } > > set_fedora() >@@ -46,8 +41,6 @@ set_fedora() > -e "s|^#baseurl=http://download.example/pub/fedora/linux|baseurl=${mirror_addr}/fedora|g" \ > -i.bak \ > /etc/yum.repos.d/fedora*.repo >- >- dnf install -y -q "${packages}" > } > > set_centos() >@@ -74,7 +67,43 @@ set_centos() > ;; > > esac >- yum install -y -q "${packages}" >+} >+ >+adapt_lkp_depends() >+{ >+ os=$(echo "${_system_name}" | tr '[:upper:]' '[:lower:]') >+ lkp_depends=$(get_dependency_packages $os lkp | xargs) >+} >+ >+get_package_manager() >+{ >+ has_cmd "yum" && installer="yum" >+ has_cmd "dnf" && installer="dnf" && return >+ has_cmd "apt-get" && installer="apt-get" && return >+ has_cmd "pacman" && installer="pacman" && return >+ has_cmd "zypper" && installer="zypper" && return >+} >+ >+install_depends() >+{ >+ packages="$1" >+ >+ get_package_manager >+ case "$installer" in >+ apt-get) >+ export DEBIAN_FRONTEND=noninteractive >+ "$installer" install -yqm ${packages} >+ ;; >+ dnf|yum) >+ "$installer" install -y -q --skip-broken ${packages} >+ ;; >+ pacman) >+ "$installer" -Sy --noconfirm --needed ${packages} >+ ;; >+ zypper) >+ "$installer" -q install -y ${packages} >+ ;; >+ esac > } > > set_local_mirror() >@@ -104,3 +133,5 @@ set_local_mirror() > } > > set_local_mirror >+adapt_lkp_depends >+install_depends "${lkp_depends}" >-- >2.23.0 >

1 0

test mail for mutt
by Dong Bingqi 08 Feb '22

08 Feb '22

test msg

1 0

test mail for mutt
by Li Yuanchao 08 Feb '22

08 Feb '22

test msg

1 0

help
by 小青 07 Feb '22

07 Feb '22

1 0

test msg to lsd 2
by Wang Yong 07 Feb '22

07 Feb '22

test msg

1 0

test mail to lsd
by Wu Zhende 07 Feb '22

07 Feb '22

test mail

1 0

test mail to wjh
by Luan Shengde 07 Feb '22

07 Feb '22

test mail

1 0

test mail to lsd
by Wang Lin 07 Feb '22

07 Feb '22

test msg

1 0

test mail to wl
by Luan Shengde 07 Feb '22

07 Feb '22

test msgg

1 0

test mail
by Wang Lin 07 Feb '22

07 Feb '22

test msg

1 0

reproducible-build TODO list
by Lu Weitao 07 Feb '22

07 Feb '22

## TODO list - TODO1.1: script support set env for build env --- compass-ci jihui - TODO1.2: script record buildinfo file in testbox --- IT/compass-ci - based on debian buildinfo - TODO1.3: 输出用户指定的息into buildinfo? new lib/buildinfo.sh --- IT func1() func2() func3() save to $result_root/buildinfo.yaml modify rpmbuild script to call the above funcs. --- compass-ci wangyong - TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao - TODO2: 比较构建结果是否相同 --- compass-ci weitao compass ci reuse data API es-find compare in client side 格式和进一步优化: IT client: ask use python? - TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT - TODO4 toolset --- IT/compass-ci option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong --------- Thanks, Lu Weitao

1 0

test mail 0207
by Lu Weitao 07 Feb '22

07 Feb '22

test msg 0207

1 0

2025

2024

2023

2022

2021

2020

Compass-ci February 2022