- Compass-ci - mailweb.openeuler.org

meeting
by Lu Weitao 24 Feb '22

24 Feb '22

aa

1 0

test mail 2
by weijihui 24 Feb '22

24 Feb '22

test msg

1 0

meeting
by Lu Weitao 22 Feb '22

22 Feb '22

aa

1 0

id
by Lu Weitao 22 Feb '22

22 Feb '22

aa

1 0

key
by Luan Shengde 21 Feb '22

21 Feb '22

1 0

test
by Luan Shengde 21 Feb '22

21 Feb '22

1 0

test mail
by Luan Shengde 21 Feb '22

21 Feb '22

1 0

Re: [PATCH v6 lkp-tests] sbin/search, sbin/select: add es search cmd
by Wu Fengguang 17 Feb '22

17 Feb '22

On Thu, Feb 17, 2022 at 02:45:04PM +0800, Wei Jihui wrote: >On Thu, Feb 17, 2022 at 12:44:43PM +0800, Wu Fengguang wrote: >>On Wed, Feb 16, 2022 at 09:10:34PM +0800, Wei Jihui wrote: >>>sbin/search is cci client cmd for query es by dsl >> >>copy/paste error > >what is the error? That's not related to SQL. The below is. >>>sbin/select is cci client cmd for query es by sql Thanks, Fengguang

1 0

test mail
by Cao Xueliang 16 Feb '22

16 Feb '22

test

1 0

test reply 8
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test reply 7
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test reply 6
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test auto reply 5
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test auto reply 4
by Luan Shengde 15 Feb '22

15 Feb '22

test msg

1 0

test mail for auto reply 3
by Luan Shengde 15 Feb '22

15 Feb '22

test mail for auto reply 3

1 0

for test auto reply
by Luan Shengde 15 Feb '22

15 Feb '22

test msg for auto reply

1 0

test mail for compass-ci.org
by Cao Xueliang 15 Feb '22

15 Feb '22

test msg

1 0

Re: [PATCH v4 compass-ci] data-api: add es search api
by weijihui 15 Feb '22

15 Feb '22

On Tue, Feb 15, 2022 at 09:35:52AM +0800, Wei Jihui wrote: >On Mon, Feb 14, 2022 at 07:21:33PM +0800, Wu Fengguang wrote: >>On Mon, Feb 14, 2022 at 06:25:41PM +0800, weijihui wrote: >>>On Fri, Feb 11, 2022 at 06:51:34PM +0800, Wu Fengguang wrote: >>>>On Thu, Feb 10, 2022 at 09:25:37PM +0800, Wei Jihui wrote: >>>>>From: Wei Jihui <weijihuiall(a)163.com> >>>>> >>>>>1. /data_api/es/:index/_search is es search by dsl >>>> >>>>>2. /data_api/es/:index/_search_by_sql is es search by sql >>>> >>>>建议用 >>>> /data_api/_opendistro/_sql >>> >>>但是现在用的不是这个接口，用的是https://github.com/NLPchina/elasticsearch-sql >> >>两个repo有继承性，基本是兼容的。 >>要么也可以用NLPchina的接口，怎么也比现在的这个好: >> >> 2. /data_api/es/:index/_search_by_sql is es search by sql > >好的，那就用这个/data_api/es/:index/_nlpcn/sql 指向原始的接口 /data_api/_nlpcn/sql sinatra match pattern /data_api/es/:index/_search sinatra match pattern /data_api/redis/xxx sinatra match pattern Thanks, Fengguang

1 0

Re: [todo] auto submit cci-makepkg for all tests
by Wu Fengguang 15 Feb '22

15 Feb '22

On Mon, Feb 14, 2022 at 04:34:20PM +0800, Cao Xueliang wrote: ># auto submit cci-makepkg before run test suite > >example: >- job.yaml >--- >'#! rpmbuild.yaml': >suite: rpmbuild >category: functional >rpmbuild: >repo_addr: https://vault.centos.org/6.10/os/Source/SPackages/ed-1.1-3.3.el6.src.rpm >arch: aarch64 >custom_repo_name: unstable/compatible/6 >mount_repo_addr: https://api.compass-ci.openeuler.org:20018/rpm/testing/openeuler-20.03-LTS-… >1/compat-centos6/standard_\$basearch/\$basearch >mount_repo_name: compatible-6 >docker_image: openeuler:20.03-LTS-SP1 >``` >- generate cci-makepkg.yaml ># detect $LKP_SRC/pkg/rpmbuild/PKGBUILD first ># detect /srv/initrd/pkg/container/openeuler/aarch64/20.03-LTS-SP1/rpmbuild/latest.cgz >``` >suite: cci-makepkg >category: functional > >cci-makepkg: > benchmark: rpmbuild # from rpmbuild.yaml >os: # from rpmbuild.yaml >os_arch: # from rpmbuild.yaml >os_mount: # from rpmbuild.yaml >``` Looks good. >- submit cci-makepkg.yaml & rpmbuild.yaml > - if cci-makepkg job success and exists latest.cgz then submit rpmbuild.yaml > - if cci-makepkg job not success then echo error & no need submit rpmbuild.yaml Just to make it clear: Here "submit cci-makepkg.yaml" should not mean "cci submit". The scheduler should - check and setup job depends - auto create and dispatch cci-makepkg.yaml - continue dispatch rpmbuild.yaml, when the above job succeeded Thanks, Fengguang

1 0

Re: [PATCH v5 compass-ci] data-api: add es search api
by Wei Jihui 15 Feb '22

15 Feb '22

On Mon, Feb 14, 2022 at 07:07:14PM +0800, Wei Jihui wrote: >From: Wei Jihui <weijihuiall(a)163.com> > pls ignore this patch. Thanks, Jihui >1. /data_api/es/:index/_search is es search by dsl >2. /data_api/es/:index/_search_by_sql is es search by sql > >Signed-off-by: Wei Jihui <weijihuiall(a)163.com> >--- > container/data-api/data-api | 36 +++++++----- > lib/es_client.rb | 13 ++-- > src/lib/data_api.rb | 93 +++++------------------------ > src/lib/data_api/es_data_api.rb | 101 ++++++++++++++++++++++++++++++++ > 4 files changed, 143 insertions(+), 100 deletions(-) > create mode 100644 src/lib/data_api/es_data_api.rb > >diff --git a/container/data-api/data-api b/container/data-api/data-api >index 2bd6e527..834ec702 100755 >--- a/container/data-api/data-api >+++ b/container/data-api/data-api >@@ -7,8 +7,6 @@ require 'sinatra' > > CCI_SRC = ENV['CCI_SRC'] || '/c/compass-ci' > require "#{CCI_SRC}/src/lib/data_api.rb" >-require "#{CCI_SRC}/src/lib/service_logs/service_logs.rb" >-require "#{CCI_SRC}/lib/es_query.rb" > > set :bind, '0.0.0.0' > set :port, 10005 >@@ -17,24 +15,32 @@ set :environment, :production > > # POST > # eg1: >-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >+# curl -X POST localhost:10005/data_api/es/jobs/_search -H 'Content-Type: application/json' -d '{ > # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >-# "index": "jobs", >-# "my_token": "16132550-...", >-# "my_account": "auto-submit", >-# "query_type": "eql" # can be omitted in this scenario >+# "cci_credentials": { >+# "my_token": "16132550-...", >+# "my_account": "auto-submit" >+# } > # }' >-# eg2: >-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >+# Response: >+# - es_result : JSON >+post '/data_api/es/:index/_search' do >+ request.body.rewind # in case someone already read it >+ es_search(params['index'], request.body.read) >+end >+ >+# POST >+# eg1: >+# curl -X POST localhost:10005/data_api/es/jobs/_search_by_sql -H 'Content-Type: application/json' -d '{ > # "query": "SELECT * FROM jobs WHERE ...", >-# "index": "jobs", >-# "my_token": "16132550-...", >-# "my_account": "auto-submit", >-# "query_type": "sql" # required >+# "cci_credentials": { >+# "my_token": "16132550-...", >+# "my_account": "auto-submit" >+# } > # }' > # Response: > # - es_result : JSON >-post '/data_api/es_find' do >+post '/data_api/es/:index/_search_by_sql' do > request.body.rewind # in case someone already read it >- es_find(request.body.read) >+ es_search_by_sql(params['index'], request.body.read) > end >diff --git a/lib/es_client.rb b/lib/es_client.rb >index 4ae1fc00..63e42d76 100644 >--- a/lib/es_client.rb >+++ b/lib/es_client.rb >@@ -6,9 +6,9 @@ require_relative 'es_query.rb' > # ------------------------------------------------------------------------------------------- > # put_source_by_id(source_id, source_content) > # - put a source to ES /<target>/_doc/<_id>, ingore the existence of source >-# >-# query_by_sql(query_sql) >-# - query es db by query_sql, query_sql is sql, like "SELECT * FROM JOBS WHERE ..." >+# >+# search_by_sql(search_sql) >+# - search es db by search_sql, search_sql is sql, like "SELECT * FROM JOBS WHERE ..." > # > # ------------------------------------------------------------------------------------------- > class ESClient < ESQuery >@@ -29,12 +29,11 @@ class ESClient < ESQuery > ) > end > >- # query es db by query_sql, query_sql is sql, like: >+ # search es db by search_sql, search_sql is sql, like: > # - "SELECT id, suite FROM JOBS" > # - "SELECT * FROM accounts WHERE my_account='test_user'" > # this plugin from: https://github.com/NLPchina/elasticsearch-sql >- def query_by_sql(query_sql) >- @client.perform_request('GET', '_nlpcn/sql', {}, query_sql) >+ def search_by_sql(search_sql) >+ @client.perform_request('GET', '_nlpcn/sql', {}, search_sql) > end >- > end >diff --git a/src/lib/data_api.rb b/src/lib/data_api.rb >index eeaf0f98..8aa4304b 100644 >--- a/src/lib/data_api.rb >+++ b/src/lib/data_api.rb >@@ -3,90 +3,27 @@ > # frozen_string_literal: true > > CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' >-require 'json' >-require 'elasticsearch' >-require 'set' >-require_relative '../../lib/constants.rb' >-require_relative '../../lib/es_query.rb' >-require_relative '../../lib/es_client.rb' >-require_relative '../../lib/json_logger.rb' >+require "#{CCI_SRC}/lib/json_logger.rb" >+require "#{CCI_SRC}/src/lib/data_api/es_data_api.rb" > >-ES_ACCOUNTS = ESQuery.new(index: 'accounts') >-REQUIRED_TOKEN_INDEX = Set.new(['jobs']) >-UNOPEN_INDEX = Set.new(['accounts']) >-ES_QUERY_KEYWORD = Set.new(['term', 'match']) >- >-def es_find(params) >+def es_search(index, params) > begin >- result = query_es(params) >+ result = EsDataApi.search(index, params) > rescue StandardError => e >- log_error({ >- 'message' => e.message, >- 'error_message' => "query es error" >- }) >- return [500, headers.merge('Access-Control-Allow-Origin' => '*'), e.message] >+ error_msg = { 'error_msg' => e.message } >+ log_error(error_msg) >+ return [200, headers.merge('Access-Control-Allow-Origin' => '*'), error_msg.to_json] > end > [200, headers.merge('Access-Control-Allow-Origin' => '*'), result.to_json] > end > >-def query_es(params) >- request_body = JSON.parse(params) >- index = request_body['index'] || 'jobs' >- query = request_body['query'] || {'query' => {}} >- return "#{index} is not opened" if UNOPEN_INDEX.any? { |unopen_index| index.include?(unopen_index) } >- >- if REQUIRED_TOKEN_INDEX.any? { |req_index| index.include?(req_index) } >- my_account = request_body['my_account'] >- my_token = request_body['my_token'] >- return "missed my_account" unless my_account >- return "missed my_token" unless my_token >- return "user authentication failed" unless verify_user(my_account, my_token) >- >- query = build_account_query(query, my_account, request_body['query_type'], index) >- end >- >- if request_body['query_type'] == 'sql' >- es = ESClient.new(index: index) >- return es.query_by_sql(query).body >- else >- es = Elasticsearch::Client.new(hosts: ES_HOSTS) >- return es.search index: index + '*', body: query >- end >-end >- >-def verify_user(my_account, my_token) >- query = {} >- query['my_account'] = my_account if my_account >- query['my_token'] = my_token if my_token >- >- result = ES_ACCOUNTS.multi_field_query(query)['hits']['hits'] >- return nil if result.size == 0 >- >- result[0]['_source']['my_token'] == my_token >-end >- >-def build_account_query(query, my_account, query_type, index) >- if query_type == 'sql' >- user_limit = "my_account='#{my_account}'" >- if query =~ /where/i >- query = query.gsub(/FROM\s*\S+/i, "FROM #{index} ") >- query = query.gsub(/where/i, "WHERE #{user_limit} AND") >- else >- query = query.gsub(/from\s*\S+/i, "FROM #{index} WHERE #{user_limit}") >- end >- else >- query['query'] ||= {} >- query['query']['bool'] ||= {} >- query['query']['bool']['must'] ||= [] >- query['query']['bool']['must'] << {'term' => {'my_account' => my_account}} >- >- query['query'].each do |k, v| >- if ES_QUERY_KEYWORD.include?(k) >- query['query']['bool']['must'] << {k => v} >- query['query'].delete(k) >- end >- end >+def es_search_by_sql(index, params) >+ begin >+ result = EsDataApi.search_by_sql(index, params) >+ rescue StandardError => e >+ error_msg = { 'error_msg' => e.message } >+ log_error(error_msg) >+ return [200, headers.merge('Access-Control-Allow-Origin' => '*'), error_msg.to_json] > end >- >- query >+ [200, headers.merge('Access-Control-Allow-Origin' => '*'), result.to_json] > end >diff --git a/src/lib/data_api/es_data_api.rb b/src/lib/data_api/es_data_api.rb >new file mode 100644 >index 00000000..ed89b596 >--- /dev/null >+++ b/src/lib/data_api/es_data_api.rb >@@ -0,0 +1,101 @@ >+# SPDX-License-Identifier: MulanPSL-2.0+ >+# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >+# frozen_string_literal: true >+ >+CCI_SRC ||= ENV['CCI_SRC'] || '/c/compass-ci' >+require 'json' >+require 'set' >+require "#{CCI_SRC}/lib/constants.rb" >+require "#{CCI_SRC}/lib/es_client.rb" >+ >+# this module is for es api >+# - search >+# - search_by_sql >+module EsDataApi >+ ES_ACCOUNTS = ESClient.new(index: 'accounts') >+ OPEN_INDEX = Set.new(['jobs']) >+ REQUIRED_TOKEN_INDEX = Set.new(['jobs']) >+ ES_QUERY_KEYWORD = Set.new(%w[term match]) >+ >+ def self.credentials_for_dsl(query, my_account) >+ query['query'] ||= {} >+ query['query']['bool'] ||= {} >+ query['query']['bool']['must'] ||= [] >+ query['query']['bool']['must'] << { 'term' => { 'my_account' => my_account } } >+ query = handle_dsl_query(query) >+ return query >+ end >+ >+ def self.handle_dsl_query(query) >+ query['query'].each do |k, v| >+ if ES_QUERY_KEYWORD.include?(k) >+ query['query']['bool']['must'] << { k => v } >+ query['query'].delete(k) >+ end >+ end >+ return query >+ end >+ >+ def self.check_my_account(request_body) >+ cci_credentials = request_body['cci_credentials'] || {} >+ my_account = cci_credentials['my_account'] >+ my_token = cci_credentials['my_token'] >+ raise 'user authentication failed, please check my_account and my_token.' unless verify_user(my_account, my_token) >+ >+ return my_account >+ end >+ >+ def self.search(index, params) >+ request_body = JSON.parse(params) >+ query = request_body['query'] || { 'query' => {} } >+ raise "#{index} is not opened for user query" unless OPEN_INDEX.include?(index) >+ >+ if REQUIRED_TOKEN_INDEX.include?(index) >+ my_account = check_my_account(request_body) >+ query = credentials_for_dsl(query, my_account) >+ end >+ es = Elasticsearch::Client.new(hosts: ES_HOSTS) >+ return es.search index: index + '*', body: query >+ end >+ >+ def self.credentials_for_sql(query, my_account, index) >+ user_limit = "my_account='#{my_account}'" >+ if query =~ /where/i >+ query = query.gsub(/FROM\s*\S+/i, "FROM #{index} ") >+ query = query.gsub(/where/i, "WHERE #{user_limit} AND") >+ else >+ query = query.gsub(/from\s*\S+/i, "FROM #{index} WHERE #{user_limit}") >+ end >+ return query >+ end >+ >+ def self.search_by_sql(index, params) >+ request_body = JSON.parse(params) >+ query = request_body['query'] >+ raise "#{index} is not opened for user query" unless OPEN_INDEX.include?(index) >+ >+ if REQUIRED_TOKEN_INDEX.include?(index) >+ my_account = check_my_account(request_body) >+ query = credentials_for_sql(query, my_account, index) >+ end >+ >+ es = ESClient.new(index: index) >+ p query >+ p 1 >+ return es.search_by_sql(query).body >+ end >+ >+ def self.verify_user(my_account, my_token) >+ raise 'missed my_account' unless my_account >+ raise 'missed my_token' unless my_token >+ >+ query = {} >+ query['my_account'] = my_account >+ query['my_token'] = my_token >+ >+ result = ES_ACCOUNTS.multi_field_query(query)['hits']['hits'] >+ return nil if result.empty? >+ >+ result[0]['_source']['my_token'] == my_token >+ end >+end >-- >2.23.0 >

1 0

Re: [PATCH v4 compass-ci] data-api: add es search api
by Wu Fengguang 15 Feb '22

15 Feb '22

On Mon, Feb 14, 2022 at 06:25:41PM +0800, weijihui wrote: >On Fri, Feb 11, 2022 at 06:51:34PM +0800, Wu Fengguang wrote: >>On Thu, Feb 10, 2022 at 09:25:37PM +0800, Wei Jihui wrote: >>>From: Wei Jihui <weijihuiall(a)163.com> >>> >>>1. /data_api/es/:index/_search is es search by dsl >> >>>2. /data_api/es/:index/_search_by_sql is es search by sql >> >>建议用 >> /data_api/_opendistro/_sql > >但是现在用的不是这个接口，用的是https://github.com/NLPchina/elasticsearch-sql 两个repo有继承性，基本是兼容的。要么也可以用NLPchina的接口，怎么也比现在的这个好: 2. /data_api/es/:index/_search_by_sql is es search by sql Thanks, Fengguang

2 1

Re: [PATCH v4 compass-ci] data-api: add es search api
by weijihui 14 Feb '22

14 Feb '22

On Fri, Feb 11, 2022 at 06:53:02PM +0800, Wu Fengguang wrote: >>+# "cci_authorization": { >>+# "my_token": "16132550-...", >>+# "my_account": "auto-submit" >>+# } > >>+# "cci_authorization": { >>+# "my_token": "16132550-...", >>+# "my_account": "auto-submit" >>+# } > >s/cci_authorization/cci_credentials/ > ok Thanks, Jihui >Thanks, >Fengguang

1 0

Re: [PATCH v4 compass-ci] data-api: add es search api
by weijihui 14 Feb '22

14 Feb '22

On Fri, Feb 11, 2022 at 06:51:34PM +0800, Wu Fengguang wrote: >On Thu, Feb 10, 2022 at 09:25:37PM +0800, Wei Jihui wrote: >>From: Wei Jihui <weijihuiall(a)163.com> >> >>1. /data_api/es/:index/_search is es search by dsl > >>2. /data_api/es/:index/_search_by_sql is es search by sql > >建议用 > /data_api/_opendistro/_sql 但是现在用的不是这个接口，用的是https://github.com/NLPchina/elasticsearch-sql Thanks, Jihui > >:index 有可能数据重复，此时不可信。 > >因为真正起作用的是 select 语句里的from， >必需解析后得到index，再基于它做认证。 > >Thanks, >Fengguang >

1 0

Re: [meeting] stats rules
by Wu Fengguang 14 Feb '22

14 Feb '22

># stats > > global stats pattern > > *.(log|message|element): > type: string - .log 出自下面的TODO - .log 命名不够清楚，是不是改成.logfile；类似的.message .element也可以改长一点 - .xxx 后缀有很小的可能与正常id混淆，改成#logfile#或者类似明显不同寻常的后缀，区分度较好，会比较安全和明显 Thanks, Fengguang --- [TODO] virttest stats TODO1 ===== (1/1) type_specific.io-github-autotest-libvirt.libvirt_usb_hotplug_device.attach_device.one_count.storage: FAIL: failed to attach device.\nDetail: command: '/bin/virsh attach-device avocado-vt-vm1 /tmp/lkp/xml_utils_temp_7xma5yyz.xml'\nexit_status: 1\nduration: 0.029788970947265625\ninterrupted: False\npid: 30074\nencoding: 'UTF-8'\nstdout: '\n'\nstderr: "error: Failed to a... (49.09 s) ==> libvirt_usb_hotplug_device.attach_device.one_count.storage.fail: 1 (for bisect) feature3.casexxx.fail: 1 TODO2 ===== for bisect report email feature3.casexxx.message => (1/1) type_specific.io-github-autotest-libvirt.libvirt_usb_hotplug_device.attach_device.one_count.storage: FAIL: failed to attach device.\nDetail: command: '/bin/virsh attach-device avocado-vt-vm1 /tmp/lkp/xml_utils_temp_7xma5yyz.xml'\nexit_status: 1\nduration: 0.029788970947265625\ninterrupted: False\npid: 30074\nencoding: 'UTF-8'\nstdout: '\n'\nstderr: "error: Failed to a... (49.09 s) TODO3 ===== for email/web report feature3.casexxx.log => job-2020-12-08T20.08-3bf7720/job.log (in result_root/) JOB LOG : /root/avocado/job-results/job-2020-12-08T20.08-3bf7720/job.log TODO4 ===== featureXXX.result.ok => featureXXX.nr_pass featureXXX.result.all => featureXXX.nr_total featureXXX.result.rate => featureXXX.pass_rate (perhaps not that useful, since the total pass_rate looks more useful, which can only be calculated by the framework across multi-jobs) RESULTS : PASS 0 | ERROR 0 | FAIL 1 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 => stats => libvirt_usb_hotplug_device.nr_pass: 0 (if 0, may not print this line) libvirt_usb_hotplug_device.nr_fail: 1 (for account report) libvirt_usb_hotplug_device.nr_error: 0 libvirt_usb_hotplug_device.nr_warn: 0 libvirt_usb_hotplug_device.nr_total: 1 Thanks, Fengguang

1 0

[meeting] stats rules
by Lu Weitao 14 Feb '22

14 Feb '22

# type fail 0/1 counter N perf 自然数，float text [string]/string datetime? epoch? number? is array or scalar? is kpi? is negative corelation? perf split to throughput, latency, power? # per-script definition etc/failure ~/lkp-tests/etc/event-counter-prefixes ~/lkp-tests/etc/independent-counter-prefixes 1|index-perf-all.yaml 2|index-perf.yaml 3|perf-metrics-patterns 4|perf-metrics-prefixes 5|perf-metrics-threshold.yaml are all global what if a user to add one single test case and script? best to only define it in a self maintained dir with yaml description tests/unixbench + tests/unixbench.yaml os_depends: ubuntu: unixbench input.params: # will be indexed by ES nr_task: description: #type: int # for lint and catch early typo at submit time unit: default: test: input.env: # won't be indexed by ES output.stats: score match by name by regex by prefix type: record: by regex start_time .*\.message result_root/stats.json result_root/record.json # stats global stats pattern *.(log|message|element): type: string script stats pattern buildinfo\..*: type: hash example proc-vmstat pattern nr_.*: type perf pg.*: counter xxx yyy zzz: counter etc/event-counter-prefixes:proc-vmstat.nr_promoted etc/event-counter-prefixes:proc-vmstat.nr_promote_fail etc/event-counter-prefixes:proc-vmstat.nr_promote_ratelimit etc/event-counter-prefixes:proc-vmstat.nr_promote_isolate_fail etc/event-counter-prefixes:proc-vmstat.allocstall etc/event-counter-prefixes:proc-vmstat.compact_ etc/event-counter-prefixes:proc-vmstat.htlb_buddy_alloc_ etc/event-counter-prefixes:proc-vmstat.kswapd_ etc/event-counter-prefixes:proc-vmstat.nr_dirtied etc/event-counter-prefixes:proc-vmstat.nr_vmscan_immediate_reclaim etc/event-counter-prefixes:proc-vmstat.nr_written etc/event-counter-prefixes:proc-vmstat.numa_ etc/event-counter-prefixes:proc-vmstat.pageoutrun etc/event-counter-prefixes:proc-vmstat.pg etc/event-counter-prefixes:proc-vmstat.pswp etc/event-counter-prefixes:proc-vmstat.slabs_scanned etc/event-counter-prefixes:proc-vmstat.thp_ etc/event-counter-prefixes:proc-vmstat.unevictable_pgs_ etc/event-counter-prefixes:proc-vmstat.zone_reclaim_failed problem: this is not clean namespace split log.xxx: xxx: solution1: move log out of stats? job.stats.buildinfo.hash from $(stats stdout), $script.yaml job.record.buildinfo.env from $script.record.yaml also job.stats.xxx.message: string/[] stats only has number types? # stats namespace change implement

1 0

[todo] auto submit cci-makepkg for all tests
by Cao Xueliang 14 Feb '22

14 Feb '22

# auto submit cci-makepkg before run test suite example: - job.yaml --- '#! rpmbuild.yaml': suite: rpmbuild category: functional rpmbuild: repo_addr: https://vault.centos.org/6.10/os/Source/SPackages/ed-1.1-3.3.el6.src.rpm arch: aarch64 custom_repo_name: unstable/compatible/6 mount_repo_addr: https://api.compass-ci.openeuler.org:20018/rpm/testing/openeuler-20.03-LTS-… 1/compat-centos6/standard_\$basearch/\$basearch mount_repo_name: compatible-6 docker_image: openeuler:20.03-LTS-SP1 ``` - generate cci-makepkg.yaml # detect $LKP_SRC/pkg/rpmbuild/PKGBUILD first # detect /srv/initrd/pkg/container/openeuler/aarch64/20.03-LTS-SP1/rpmbuild/latest.cgz ``` suite: cci-makepkg category: functional cci-makepkg: benchmark: rpmbuild # from rpmbuild.yaml os: # from rpmbuild.yaml os_arch: # from rpmbuild.yaml os_mount: # from rpmbuild.yaml ``` - submit cci-makepkg.yaml & rpmbuild.yaml - if cci-makepkg job success and exists latest.cgz then submit rpmbuild.yaml - if cci-makepkg job not success then echo error & no need submit rpmbuild.yaml

1 0

Re: [PATCH v3 compass-ci] data-api: add es search api
by Wu Fengguang 11 Feb '22

11 Feb '22

On Wed, Feb 09, 2022 at 09:23:20PM +0800, weijihui wrote: >On Wed, Feb 09, 2022 at 01:21:14PM +0800, Wu Fengguang wrote: >>On Tue, Feb 08, 2022 at 03:10:57PM +0800, Wei Jihui wrote: >>>From: Wei Jihui <weijihuiall(a)163.com> >>> >>>1. /data_api/es/:index/es_search is es search by dsl >> >>是这样的吧： >> >> es_search => _search > >yes > >> >>>2. /data_api/es/:index/es_search_by_sql is es search by sql >> >>我看你后面用了 >> >> >+# curl -X POST localhost:10005/data_api/es/jobs/_search_by_sql >> >>然后调用的是 >> >> >+ @client.perform_request('GET', '_nlpcn/sql', {}, search_sql) >> >>但我们应该最终调用这个吧: >> >> https://github.com/opendistro-for-elasticsearch/sql >> >> POST https://<host>:<port>/_opendistro/_sql >> { >> "query": "SELECT * FROM my-index LIMIT 50" >> } >> >>相应的API是不是相应的写成 >> >> /data_api/_opendistro/_sql > >这个接口需要安装的插件没有我们用的es版本 >目前用的这个接口 >_nlpcn/sql 可以计划下升级到 _opendistro 和与之匹配的ES。 API层面，可以先用 _opendistro 免得折腾。 >> >>>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>>+# curl -X POST localhost:10005/data_api/es/jobs/_search -H 'Content-Type: application/json' -d '{ >>> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >>>-# "index": "jobs", >>>-# "my_token": "16132550-...", >>>-# "my_account": "auto-submit", >>>-# "query_type": "eql" # can be omitted in this scenario >>>+# "cci_authorization": { >> >>这个貌似不错： >> >> credentials > >cci_credentials? Yes. Thanks, Fengguang

1 0

Re: [PATCH todo 1/2] add todo for docker improvement
by Luan Shengde 11 Feb '22

11 Feb '22

pls ignore this mail Thanks, Luan Shengde On Fri, Feb 11, 2022 at 05:05:52PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > .../todo/20220216-5d:docker-improvement.md | 35 +++++++++++++++++++ > 1 file changed, 35 insertions(+) > create mode 100644 people/LuanShengde/todo/20220216-5d:docker-improvement.md > >diff --git a/people/LuanShengde/todo/20220216-5d:docker-improvement.md b/people/LuanShengde/todo/20220216-5d:docker-improvement.md >new file mode 100644 >index 00000000..416c749d >--- /dev/null >+++ b/people/LuanShengde/todo/20220216-5d:docker-improvement.md >@@ -0,0 +1,35 @@ >+# docker improvement >+ >+## handle download failure and check dir besides /lkp and /opt >+ >+original: >+ curl_cmd: %x(curl -sS --create-dirs -o #{path}/#{name} #{url} && gzip -dc #{path}/#{name} | cpio -idu -D #{path}) >+ download cgz file and unpack it in one line >+ >+improvement: >+ curl_cmd: split download cgz file, check dir and unpack the cgz file into standalone lines. >+ system('wget -t 3 -T 20 -O #{path}/#{name} #{url} 2> /dev/null') >+ use system to call the linux command, it will retusn true/false >+ - true: execute success >+ - false: execute in failure >+ raise error if download in failure >+ >+ %x(gzip -dc path/cgz_file | cpio -idt) >+ use %x(command) can get the output of the command, extract the file list of the cgz file >+ check files from the file list and check if the file/dir is besides the /lkp or /opt >+ >+ if no files/dirs besides /lkp or /opt >+ %x(gzip -dc path/cgz_file | cpio -idu -D path) >+ >+ if exist files/dirs besides /lkp or /opt >+ save the cgz to opt >+ unpack the cgz file in container >+ >+ --- >+ maybe can ignore check the cgz file if it containers files/dirs besides /lkp or /opt >+ just unpack all the cgz file in the container >+ --- >+ >+## /usr/local/bin/busybox should be ro >+ >+## cancel mount unnecessary dir >-- >2.23.0 >

1 0

Re: [PATCH v2 compass-ci] assign-account-tool: update email content
by Luan Shengde 11 Feb '22

11 Feb '22

On Thu, Feb 10, 2022 at 09:53:48PM +0800, Wei Jihui wrote: >On Thu, Feb 10, 2022 at 03:58:43PM +0800, Luan Shengde wrote: >>From: Luan Shengde <shdluan(a)163.com> >> >>for only login with public key is allow now, update the email content. >> >>Signed-off-by: Luan Shengde <shdluan(a)163.com> >>--- >> .../assign-account/build-send-account-email.rb | 13 ++++++++++--- >> container/assign-account/build-update-email.rb | 13 +++++++++++-- >> 2 files changed, 21 insertions(+), 5 deletions(-) >> >>diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb >>index 3c414041..f30de78b 100755 >>--- a/container/assign-account/build-send-account-email.rb >>+++ b/container/assign-account/build-send-account-email.rb >>@@ -16,10 +16,17 @@ def build_message(email, account_info) >> Login command: >> ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} >> >>- Account password: >>- #{account_info['my_password']} >>+ You can directly use the public key you offered to login the account-ver server. >> >>- You can use your pub_key to login if you have offered one. >>+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >>+ >>+ name: Zhang Dewan >>+ phone: 18969058870 >>+ email: zhangdewan1(a)compass-ci.org >>+ >>+ name: Luan Shengde >>+ phone: 15109218229 >>+ email: luanshengde(a)compass-ci.org >> >> Suggest: >> If you use the password to login, change it in time. >>diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb >>index 40d6b3b3..0a298ac8 100755 >>--- a/container/assign-account/build-update-email.rb >>+++ b/container/assign-account/build-update-email.rb >>@@ -18,8 +18,17 @@ def build_update_message(email, account_info, conf_info) >> Login command: >> ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} >> >>- Account password: >>- #{account_info['my_password']} >>+ You can directly use the public key you offered to loging the jumper server. > > loging --> login? > > Thanks, > Jihui OK, I will fix it Thanks, Luan Shengde > > >>+ >>+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >>+ >>+ name: Zhang Dewan >>+ phone: 18969058870 >>+ email: zhangdewan1(a)compass-ci.org >>+ >>+ name: Luan Shengde >>+ phone: 15109218229 >>+ email: luanshengde(a)compass-ci.org >> >> regards >> compass-ci >>-- >>2.23.0 >>

1 0

Re: [PATCH v2 compass-ci] assign-account-tool: update email content
by weijihui 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 03:58:43PM +0800, Luan Shengde wrote: >From: Luan Shengde <shdluan(a)163.com> > >for only login with public key is allow now, update the email content. > >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > .../assign-account/build-send-account-email.rb | 13 ++++++++++--- > container/assign-account/build-update-email.rb | 13 +++++++++++-- > 2 files changed, 21 insertions(+), 5 deletions(-) > >diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb >index 3c414041..f30de78b 100755 >--- a/container/assign-account/build-send-account-email.rb >+++ b/container/assign-account/build-send-account-email.rb >@@ -16,10 +16,17 @@ def build_message(email, account_info) > Login command: > ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > >- Account password: >- #{account_info['my_password']} >+ You can directly use the public key you offered to login the account-ver server. > >- You can use your pub_key to login if you have offered one. >+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >+ >+ name: Zhang Dewan >+ phone: 18969058870 >+ email: zhangdewan1(a)compass-ci.org >+ >+ name: Luan Shengde >+ phone: 15109218229 >+ email: luanshengde(a)compass-ci.org > > Suggest: > If you use the password to login, change it in time. >diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb >index 40d6b3b3..0a298ac8 100755 >--- a/container/assign-account/build-update-email.rb >+++ b/container/assign-account/build-update-email.rb >@@ -18,8 +18,17 @@ def build_update_message(email, account_info, conf_info) > Login command: > ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > >- Account password: >- #{account_info['my_password']} >+ You can directly use the public key you offered to loging the jumper server. loging --> login? Thanks, Jihui >+ >+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >+ >+ name: Zhang Dewan >+ phone: 18969058870 >+ email: zhangdewan1(a)compass-ci.org >+ >+ name: Luan Shengde >+ phone: 15109218229 >+ email: luanshengde(a)compass-ci.org > > regards > compass-ci >-- >2.23.0 >

1 0

Re: reproducible-build TODO list
by weijihui 10 Feb '22

10 Feb '22

On Wed, Feb 09, 2022 at 10:35:20AM +0800, Lu Weitao wrote: >On Tue, Feb 08, 2022 at 08:20:37PM +0800, Wei Jihui wrote: >>On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >>>## TODO list >>>- TODO1.1: script support set env for build env --- compass-ci jihui >> >>what should i do for this? >> > >user submit rpmbuild.yaml will with some build params like: >fs: >kernel: >reproducible_build_env: > hostname: > datetime: >rpmbuild: > new_param: > >we should make sure these params take effect in testbox no need reproducible_build_env, just write it like runtime. runtime: 1d hostname: cc datetime: bb Thanks, Jihui > > > >--------- >Thanks, >Lu Weitao > >- > >>Thanks, >>Jihui >> >> >>> >>>- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci >>> - based on debian buildinfo >>> - TODO1.3: 输出用户指定的息into buildinfo? >>> new lib/buildinfo.sh --- IT >>> func1() >>> func2() >>> func3() >>> save to $result_root/buildinfo.yaml >>> >>> modify rpmbuild script to call the above funcs. --- compass-ci wangyong >>> >>>- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao >>> >>>- TODO2: 比较构建结果是否相同 --- compass-ci weitao >>> compass ci >>> >>> reuse data API es-find >>> compare in client side >>> >>> 格式和进一步优化: IT >>> client: ask use python? >>> >>>- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT >>> >>>- TODO4 toolset --- IT/compass-ci >>> option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 >>> option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT >>> option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong >>> >>> >>> >>>--------- >>>Thanks, >>>Lu Weitao >>>

1 0

Re: [TODO] draft 2022 targets
by Bai Jing 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 04:14:55PM +0800, Wu Fengguang wrote: >One more: > >Setup and run syzkaller, keep trigger kernel bugs and reproduce program >accumulate and reuse the above reproduce programs as kernel test case > >Thanks, >Fengguang > >On Wed, Feb 09, 2022 at 05:19:14PM +0800, Wu Fengguang wrote: >>众智项目候选 >>界面 todo owner >>- web interface luweitao >>- maintain dashboard zhangdewan >>加包 >>- OS package name mapping wuzhende >>- auto convert 3w archlinux packages wangyong >>- auto create 3w C/C++ spec liping >>- test 10w pypi upstream packages liyuanchao >>- rpm spec customize options from gentoo and ./configure output parsing >>nix >>- nix 1w upstream project tests >>- openeuler 3k overlay on nixpkgs >>- openEuler+Nix+musl docker images for popular apps, replacing alpine >>改进? >>- k8s integration luanshengde >>- selftest liuyinsi >>- bisect caoxueliang >> >>OS大赛项目候选 >>- YAML+python overlays merge tool >> >>Thanks, >>Fengguang >>

1 0

腾讯企业邮箱自动转发验证邮件
by luanshengde 10 Feb '22

10 Feb '22

亲爱的朋友：您好，腾讯企业邮箱用户 luanshengde(a)compass-ci.org 在邮箱中设置了将邮件转发至您的邮箱中。为避免不友善的转发给您带来骚扰，该设置需要您允许才能生效。如果您允许该转发设置，请点击以下链接：接受该转发请求   如果您点击上述链接无效，您可能处于纯文本模式下。请复制以下地址至浏览器地址栏进行访问： http://exmail.qq.com/cgi-bin/attrset?fun=autoreply&t=verifyfw_result&alias=…     腾讯企业邮箱腾讯企业邮箱门户：http://exmail.qq.com 企业邮箱帮助中心：http://service.exmail.qq.com

1 0

test mail for forward
by Cao Xueliang 10 Feb '22

10 Feb '22

test mail for forward

1 0

Re: [TODO] draft 2022 targets
by Bai Jing 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 04:14:55PM +0800, Wu Fengguang wrote: >One more: > >Setup and run syzkaller, keep trigger kernel bugs and reproduce program >accumulate and reuse the above reproduce programs as kernel test case > >Thanks, >Fengguang > >On Wed, Feb 09, 2022 at 05:19:14PM +0800, Wu Fengguang wrote: >>众智项目候选 >>界面 todo owner >>- web interface luweitao >>- maintain dashboard zhangdewan >>加包 >>- OS package name mapping wuzhende >>- auto convert 3w archlinux packages wangyong >>- auto create 3w C/C++ spec liping >>- test 10w pypi upstream packages liyuanchao >>- rpm spec customize options from gentoo and ./configure output parsing >>nix >>- nix 1w upstream project tests >>- openeuler 3k overlay on nixpkgs >>- openEuler+Nix+musl docker images for popular apps, replacing alpine >>改进? >>- k8s integration luanshengde >>- selftest liuyinsi >>- bisect caoxueliang >> >>OS大赛项目候选 >>- YAML+python overlays merge tool >> >>Thanks, >>Fengguang >>

1 0

Re: extract buildinfo
by Lu Weitao 10 Feb '22

10 Feb '22

please help review this design --------- Thanks, Lu Weitao On Wed, Feb 09, 2022 at 05:20:12PM +0800, Lu Weitao wrote: ># extract buildinfo and save it to ES > todo from reproducible-build flow > > ``` > after build rpm, and upload buildinfo.yaml, rpmbuild to server > | > v > $result_root/rpmbuild.yaml ---> rpmbuild.json ---> merge to stats.json > $result_root/rpmbuild_info.yaml ---> rpmbuild_info.json ---> merge to stats.json > | > v > save stats.json to ES db # 为compare， diifoscope 提供数据源 > eg: > { > "id": xxx, > "os": "openeuler", > "arch": "aarch64" > "stats": { > "rpmbuild.hash_value": xxx, > ``` >## TODO1: improve compass-ci/sbin/result2stats, extract-stats service call it > support extract buildinfo > usage: result2stats $result_root > ``` > ... > $result_root/rpmbuild ---> rpmbuild.json ---> merge to stats.json > $result_root/buildinfo.yaml ---> buildinfo.json ---> merge to stats.json > ... > ``` >### input: result_root/buildinfo(buildinfo.yaml) >- e.g. samba_4.9.5+dfsg-5+deb10u2_arm64.buildinfo >``` >Format: 1.0 >Source: samba >Binary: samba samba-libs samba-common samba-common-bin smbclient samba-testsuite registry-tools samba-dev python-samba samba-dsdb-modules samba-vfs-modules libsmbclient libsmbclient-dev winbind libpam-winbind libnss-winbind libwbclient0 libwbclient-dev ctdb >Architecture: all arm64 source >Version: 2:4.9.5+dfsg-5+deb10u2 >Checksums-Md5: > 79ffac5466e2b36633e0f429644407d0 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > d523774b75cc4d31dce4696d41d68b8f 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Checksums-Sha1: > b8c10499c1b6b7c5111c7f2d8d617ed8a2ff938e 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > 1569bfdb4399f92852863f67153cce046a0cbb38 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Checksums-Sha256: > c9094fcc63930f665707c596134ea64e44d433d3882b3de544c2a83f574aafb9 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > 3304c2fa4f7204682747bed70f7ce61b134b21f063f9ebab0aeb2e3af645b744 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Build-Origin: Debian >Build-Architecture: arm64 >Build-Date: Sun, 30 Jan 2022 09:27:50 +0000 >Installed-Build-Depends: > adduser (= 3.118), > zlib1g-dev (= 1:1.2.11.dfsg-1) > ... >Environment: > DEB_BUILD_OPTIONS="parallel=128" > SOURCE_DATE_EPOCH="1638005690" >``` >### output result_root/buildinfo.json >``` >{ > "Hash": "SHA512", > "Format": 1.0, > "Build-Architecture": "i386", > "Source": "fweb (1.62-12)", > "Binary": "fweb fweb-doc", > "Architecture": "all i386", > "Version": "1.62-12+b2", > "Binary-Only-Changes": > "fweb (1.62-12+b2) sid; urgency=low, binary-only=yes > . > * Binary-only non-maintainer upload for amd64; no source changes. > * Rebuild for multiarch sync > . > -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms(a)buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000", > ... >} >``` > > >## TODO2: save buildinfo to job(ES) > extract-service load $result_root/buildinfo.json > >- option1: merge job['buildinfo'](job['record']['buildinfo']) > meeting record: > job.record.build.xxx: v => possible solution option1 > 通用性好（job['record']['buildinfo'][xxx]） > 结构较复杂 > > job.info.build.xxx: v > 效果同上 > value type: > - output binary hash > - info (xxx.rpm) > > job['buildinfo'] > 结构简单，但只适用与构建任务，目前未发现其它类型的job有这种需求，如需使用buildinfo进行搜索，那这种方式更简单 > > **job like:** > ``` > { > "id": xxx, > "os": "openeuler", > ... > "buildinfo": { > "Hash": "SHA512", > "Format": 1.0, > ... > } > } > ``` > **advantage:** > 1. reuse job['buildinfo'] to submit again # we can also merge it to job.yaml > 2. 3rd tool such as: `diffoscope` get job['buildinfo'] > **disadvantage:** > 1. compare need get metric from buildinfo in addition > >- option2: merge buildinfo to job['stats'], job like: > ``` > { > "id": xxx, > "os": "openeuler", > "stats": { > "buildinfo.Hash": "SHA512", > "buildinfo.Format": 1.0, > ... > } > } > ``` > advantage / disadvantage: opposite of option1 > > job.stats.rpmbuild.xxx > job['stats']['rpmbuild.xxx'] > compare 不好区分buildinfo 和 rpmbuild > 需要变动lkp-tests/stats/rpmbuild，且buildinfo > result_root/rpmbuild > > job.stats.buildinfo.xxx => possible solution option2 > job['stats']['buildinfo.xxx'] > extract-stats 已经支持，无需改动代码 > 仅仅将buildinfo信息输出未 result_root/buildinfo.yaml > > compare 性能结果时忽略 buildinfo （可以新加perf-metric-ignore规则） > > job.stats.message.xxx / job.stats.xxx.message > 目前.message, .log 已经是perf-metric-ignore规则，可以考虑将buildinfo也加进去 > > > job.stats.info.build.buildinfo.xhhhhhxx => possible solution > 通用性更好，相当于对metric有个全面的分类 > 结构较多 > > > job.buildinfo.xxx（in option1） > >- option3: option1 + option2 > job like: > ``` > { > "id": xxx, > "os": "openeuler", > ... > "buildinfo": { > "Hash": "SHA512", > "Format": 1.0, > ... > }, > "stats": { > "buildinfo.Hash": "SHA512", > "buildinfo.Format": 1.0, > ... > } > } > > ``` > **advantage: opton1 + option2 ** > **disadvantage:** > 1. job bigger, --> save DB > 2. extract-service does one more thing >_______________________________________________ >Compass-ci mailing list -- compass-ci(a)openeuler.org >To unsubscribe send an email to compass-ci-leave(a)openeuler.org

1 0

Re: [PATCH v7 compass-ci] for test
by Luan Shengde 10 Feb '22

10 Feb '22

Current branch master is up to date. TO team leader: caoxueliang(a)compass-ci.org weijihui(a)compass-ci.org wufengguang(a)compass-ci.org --to=luanshengde(a)compass-ci.org TO file committer: luanshengde(a)compass-ci.org CC file author: luanshengde(a)compass-ci.org git send-email -q --no-thread --confirm=never --cc=all --cc=compass-ci(a)openeuler.org --to=caoxueliang(a)compass-ci.org --to=weijihui(a)compass-ci.org --to=wufengguang(a)compass-ci.org --to=luanshengde(a)compass-ci.org - -cc=luanshengde(a)compass-ci.org 0001-for-test-v8.patch On Thu, Feb 10, 2022 at 03:39:52PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <luanshengde(a)compass-ci.org> >--- > aaa | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 aaa > >diff --git a/aaa b/aaa >new file mode 100644 >index 00000000..4aed24e8 >--- /dev/null >+++ b/aaa >@@ -0,0 +1 @@ >+test msg >-- >2.23.0 >

1 0

Re: [PATCH v7 compass-ci] for test
by Luan Shengde 10 Feb '22

10 Feb '22

ignore this mail On Thu, Feb 10, 2022 at 03:39:52PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <luanshengde(a)compass-ci.org> >--- > aaa | 1 + > 1 file changed, 1 insertion(+) > create mode 100644 aaa > >diff --git a/aaa b/aaa >new file mode 100644 >index 00000000..4aed24e8 >--- /dev/null >+++ b/aaa >@@ -0,0 +1 @@ >+test msg >-- >2.23.0 >

1 0

Re: [meeting] install-depends summary & TODO list
by Wang Yong 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 11:22:30AM +0800, Wang Yong wrote: ># direct install depends > >- submit job.yaml >+ add a field to job.yaml from distro/depends/$job >fio: tests/ >fs: setup/ > install_depends: # add to setup/ > fs: pkg1 pkg2 > host-info: pkg3 > $script: pkg1 > >+ change pkgs' name from distro/adaptation/$distro >+ export $install_depends_packages to install_depends > >- setup/install_depends: > install packages due to $packages from YAML > >leftover: > - CCI will check, download, install initrd_deps always > > delete all deps will fix it > - share_debian_package ( run-ipconfig ) > > not work anymore > commit 8a8a57d7a166687bad1cd11e880a36c80e2c403b > > > adapt from debian to $os > ~/lkp-tests/distro/depends/$script (debian) > => > ~/lkp-tests/distro/depends/debian/$script > ~/lkp-tests/distro/adaptation/$os > > bottom line: on error, user is able to fix it by specifying exact dependencies in a depends/ file > >x * y >x y1 y2 y3 y4 > 1 2 11 12 > > find package in os similarity order > debian-11/$script > debian/$script > deb os (if os is ubuntu) > show warning on submit > > ubuntu user: lib-dev > > ~/lkp-tests/distro/depends/$os/$script (user development os) > job: $os no problem > > job: $other_os auto convert in > - submit > - scheduler > call service API: convert package name from one os to another os > >TODO: > - add filed into job.yaml with all setup/tests script --Zhende > '#! /home/wangyong/lkp-tests/jobs/host-info.yaml': > suite: host-info > category: functional > is_send_mail: false > host-info: > os: openeuler > os_arch: aarch64 > os_version: "20.03" > install_depends_pakeages: > fs: pkg1 pkg2 > host-info: pkg3 pkg4 > - convert debian/$script packages into target $os packages --Zhende > adapt from debian to $os > ~/lkp-tests/distro/depends/$script (debian) > => > ~/lkp-tests/distro/depends/debian/$script > ~/lkp-tests/distro/adaptation/$os > - copy /depends/$script to /depends/debian/$script --Wangyong > - move setup/install_depends from run_job to lkp-tests frame --Wangyong > - to prevent install depends after setup initialization > - similarity tree to list most similar rpm/deb to install --Zhende >_______________________________________________ >Compass-ci mailing list -- compass-ci(a)openeuler.org >To unsubscribe send an email to compass-ci-leave(a)openeuler.org

1 0

[meeting] install-depends summary & TODO list
by Wang Yong 10 Feb '22

10 Feb '22

# direct install depends - submit job.yaml + add a field to job.yaml from distro/depends/$job fio: tests/ fs: setup/ install_depends: # add to setup/ fs: pkg1 pkg2 host-info: pkg3 $script: pkg1 + change pkgs' name from distro/adaptation/$distro + export $install_depends_packages to install_depends - setup/install_depends: install packages due to $packages from YAML leftover: - CCI will check, download, install initrd_deps always > delete all deps will fix it - share_debian_package ( run-ipconfig ) > not work anymore commit 8a8a57d7a166687bad1cd11e880a36c80e2c403b adapt from debian to $os ~/lkp-tests/distro/depends/$script (debian) => ~/lkp-tests/distro/depends/debian/$script ~/lkp-tests/distro/adaptation/$os bottom line: on error, user is able to fix it by specifying exact dependencies in a depends/ file x * y x y1 y2 y3 y4 1 2 11 12 find package in os similarity order debian-11/$script debian/$script deb os (if os is ubuntu) show warning on submit ubuntu user: lib-dev ~/lkp-tests/distro/depends/$os/$script (user development os) job: $os no problem job: $other_os auto convert in - submit - scheduler call service API: convert package name from one os to another os TODO: - add filed into job.yaml with all setup/tests script --Zhende '#! /home/wangyong/lkp-tests/jobs/host-info.yaml': suite: host-info category: functional is_send_mail: false host-info: os: openeuler os_arch: aarch64 os_version: "20.03" install_depends_pakeages: fs: pkg1 pkg2 host-info: pkg3 pkg4 - convert debian/$script packages into target $os packages --Zhende adapt from debian to $os ~/lkp-tests/distro/depends/$script (debian) => ~/lkp-tests/distro/depends/debian/$script ~/lkp-tests/distro/adaptation/$os - copy /depends/$script to /depends/debian/$script --Wangyong - move setup/install_depends from run_job to lkp-tests frame --Wangyong - to prevent install depends after setup initialization - similarity tree to list most similar rpm/deb to install --Zhende

1 0

Re: [PATCH v3 compass-ci] data-api: add es search api
by weijihui 09 Feb '22

09 Feb '22

On Wed, Feb 09, 2022 at 01:21:14PM +0800, Wu Fengguang wrote: >On Tue, Feb 08, 2022 at 03:10:57PM +0800, Wei Jihui wrote: >>From: Wei Jihui <weijihuiall(a)163.com> >> >>1. /data_api/es/:index/es_search is es search by dsl > >是这样的吧： > > es_search => _search yes > >>2. /data_api/es/:index/es_search_by_sql is es search by sql > >我看你后面用了 > > >+# curl -X POST localhost:10005/data_api/es/jobs/_search_by_sql > >然后调用的是 > > >+ @client.perform_request('GET', '_nlpcn/sql', {}, search_sql) > >但我们应该最终调用这个吧: > > https://github.com/opendistro-for-elasticsearch/sql > > POST https://<host>:<port>/_opendistro/_sql > { > "query": "SELECT * FROM my-index LIMIT 50" > } > >相应的API是不是相应的写成 > > /data_api/_opendistro/_sql 这个接口需要安装的插件没有我们用的es版本目前用的这个接口 _nlpcn/sql > >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es/jobs/_search -H 'Content-Type: application/json' -d '{ >> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >>-# "index": "jobs", >>-# "my_token": "16132550-...", >>-# "my_account": "auto-submit", >>-# "query_type": "eql" # can be omitted in this scenario >>+# "cci_authorization": { > >这个貌似不错： > > credentials cci_credentials? Thanks, Jihui > > 1 ~ (for/as sth); ~ (to do sth) qualities, achievements, etc that make one suitable; qualifications （学历、资历等的）资格: She has the perfect credentials for the job. 她做这工作完全够格. > 2 documents showing that a person is what he claims to be, is trustworthy, etc （某人合乎条件、可以信任等的）证明书, 证件: I examined his credentials. 我查验了他的证件. > >相比而言，这更像是一个结果： > > authorization > n. 授权, 许可 > >Thanks, >Fengguang >

1 0

Re: [TODO] provide LKP dependencies in container with busybox+curl
by Luan Shengde 09 Feb '22

09 Feb '22

- use debian OS as host OS - make sure host installed busybox-static package and downloaded static-curl - mount busybox/curl into docker - create busybox symlinks in docker mkdir /opt/busybox/bin/ cd /opt/busybox/bin/ for c in wget cpio do ln -s /path/to/busybox $c done PATH=$PATH:/opt/busybox/bin/ by fengguang thanks - download .cgz depends inside docker in LKP framework - extract .cgz On Mon, Feb 07, 2022 at 02:26:54PM +0800, Wu Fengguang wrote: >Currently /c/compass-ci/providers/docker/run.sh has extra dependencies >that should be simplified. > >TODO1: install busybox in the host (I see no script to make sure it's >installed in the host). If the host OS does not has the busybox >package, it's fine to download pre-built busybox-static binary > >TODO2: make sure busybox symlinks are created in a dir like >/opt/busybox/bin/ inside the container, and _append_ it to PATH. > >TODO3: install curl-static in the host and mount into container >/opt/curl/bin/ > > https://github.com/moparisthebest/static-curl > > Grab the latest release from one of these links, by CPU architecture: > curl-amd64 > curl-i386 > curl-aarch64 > curl-armv7 > curl-ppc64le > >busybox+curl should provide all the *required* dependencies for >running LKP framework. > >Thanks, >Fengguang > >On Tue, Jan 04, 2022 at 10:07:08AM +0800, Wu Fengguang wrote: >>Let's cut down dependencies in /c/lkp-tests/distro/depends/lkp >> >>For the depends added by >> >> commit 54e515a10156080bfabc8d880b3f704b7c29a702 >> Author: Wei Jihui <weijihuiall(a)163.com> >> AuthorDate: Mon Nov 8 09:59:15 2021 +0800 >> Commit: Wei Jihui <weijihuiall(a)163.com> >> CommitDate: Thu Nov 11 14:25:03 2021 +0800 >> >> distro/depends/lkp: move cci-depends'depends to lkp's depends >> >>we can use either >>- busybox ftpput >>- rsync >>- lftp >>for uploading files. Those packages are rarely depended by normal >>packages. In comparison, curl is depended by hundreds of other >>packages in Debian. >> >>If use busybox, lkp can remove lots of dependencies. >> >>However for remote testbox, its 'busybox ftpput' command may not work well >>- need PASSIVE mode in order to go through firewall >>- not secure over Internet >>- some firewall may bann insecure service/ports >>- not good to block testbox in slow/unreliable Internet connection >> >>The same for rsync upload, if ssh is not used together. >> >>But shall we upload all remote testbox results over Internet? >>I guess not -- it won't scale. >> >>Here is a flexible upload model >> >>testbox >>-> on job finish, tar/cpio all results into a single file >> (to avoid complex mirror function for ftpput) >>-> upload via any of ftp/rsync/https to a dedicated and temp server storage space >> (eg. /srv/result/upload/$jobid/ with md5sum verification) >> (proftpd: configure as upload only dir, disable read and list-dir) >>-> tell scheduler that the job is complete >>-> run a service to unpack a job's results into final result_root >> >>remote testbox >>-> results uploaded to a local server storage >>-> local storage server runs a service to send results to central lab storage >> when asked by end user >> >>Thanks, >>Fengguang

1 0

extract buildinfo
by Lu Weitao 09 Feb '22

09 Feb '22

# extract buildinfo and save it to ES todo from reproducible-build flow ``` after build rpm, and upload buildinfo.yaml, rpmbuild to server | v $result_root/rpmbuild.yaml ---> rpmbuild.json ---> merge to stats.json $result_root/rpmbuild_info.yaml ---> rpmbuild_info.json ---> merge to stats.json | v save stats.json to ES db # 为compare， diifoscope 提供数据源 eg: { "id": xxx, "os": "openeuler", "arch": "aarch64" "stats": { "rpmbuild.hash_value": xxx, ``` ## TODO1: improve compass-ci/sbin/result2stats, extract-stats service call it support extract buildinfo usage: result2stats $result_root ``` ... $result_root/rpmbuild ---> rpmbuild.json ---> merge to stats.json $result_root/buildinfo.yaml ---> buildinfo.json ---> merge to stats.json ... ``` ### input: result_root/buildinfo(buildinfo.yaml) - e.g. samba_4.9.5+dfsg-5+deb10u2_arm64.buildinfo ``` Format: 1.0 Source: samba Binary: samba samba-libs samba-common samba-common-bin smbclient samba-testsuite registry-tools samba-dev python-samba samba-dsdb-modules samba-vfs-modules libsmbclient libsmbclient-dev winbind libpam-winbind libnss-winbind libwbclient0 libwbclient-dev ctdb Architecture: all arm64 source Version: 2:4.9.5+dfsg-5+deb10u2 Checksums-Md5: 79ffac5466e2b36633e0f429644407d0 3211 samba_4.9.5+dfsg-5+deb10u2.dsc d523774b75cc4d31dce4696d41d68b8f 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Checksums-Sha1: b8c10499c1b6b7c5111c7f2d8d617ed8a2ff938e 3211 samba_4.9.5+dfsg-5+deb10u2.dsc 1569bfdb4399f92852863f67153cce046a0cbb38 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Checksums-Sha256: c9094fcc63930f665707c596134ea64e44d433d3882b3de544c2a83f574aafb9 3211 samba_4.9.5+dfsg-5+deb10u2.dsc 3304c2fa4f7204682747bed70f7ce61b134b21f063f9ebab0aeb2e3af645b744 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Build-Origin: Debian Build-Architecture: arm64 Build-Date: Sun, 30 Jan 2022 09:27:50 +0000 Installed-Build-Depends: adduser (= 3.118), zlib1g-dev (= 1:1.2.11.dfsg-1) ... Environment: DEB_BUILD_OPTIONS="parallel=128" SOURCE_DATE_EPOCH="1638005690" ``` ### output result_root/buildinfo.json ``` { "Hash": "SHA512", "Format": 1.0, "Build-Architecture": "i386", "Source": "fweb (1.62-12)", "Binary": "fweb fweb-doc", "Architecture": "all i386", "Version": "1.62-12+b2", "Binary-Only-Changes": "fweb (1.62-12+b2) sid; urgency=low, binary-only=yes . * Binary-only non-maintainer upload for amd64; no source changes. * Rebuild for multiarch sync . -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms(a)buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000", ... } ``` ## TODO2: save buildinfo to job(ES) extract-service load $result_root/buildinfo.json - option1: merge job['buildinfo'](job['record']['buildinfo']) meeting record: job.record.build.xxx: v => possible solution option1 通用性好（job['record']['buildinfo'][xxx]）结构较复杂 job.info.build.xxx: v 效果同上 value type: - output binary hash - info (xxx.rpm) job['buildinfo'] 结构简单，但只适用与构建任务，目前未发现其它类型的job有这种需求，如需使用buildinfo进行搜索，那这种方式更简单 **job like:** ``` { "id": xxx, "os": "openeuler", ... "buildinfo": { "Hash": "SHA512", "Format": 1.0, ... } } ``` **advantage:** 1. reuse job['buildinfo'] to submit again # we can also merge it to job.yaml 2. 3rd tool such as: `diffoscope` get job['buildinfo'] **disadvantage:** 1. compare need get metric from buildinfo in addition - option2: merge buildinfo to job['stats'], job like: ``` { "id": xxx, "os": "openeuler", "stats": { "buildinfo.Hash": "SHA512", "buildinfo.Format": 1.0, ... } } ``` advantage / disadvantage: opposite of option1 job.stats.rpmbuild.xxx job['stats']['rpmbuild.xxx'] compare 不好区分buildinfo 和 rpmbuild 需要变动lkp-tests/stats/rpmbuild，且buildinfo > result_root/rpmbuild job.stats.buildinfo.xxx => possible solution option2 job['stats']['buildinfo.xxx'] extract-stats 已经支持，无需改动代码仅仅将buildinfo信息输出未 result_root/buildinfo.yaml compare 性能结果时忽略 buildinfo （可以新加perf-metric-ignore规则） job.stats.message.xxx / job.stats.xxx.message 目前.message, .log 已经是perf-metric-ignore规则，可以考虑将buildinfo也加进去 job.stats.info.build.buildinfo.xhhhhhxx => possible solution 通用性更好，相当于对metric有个全面的分类结构较多 job.buildinfo.xxx（in option1） - option3: option1 + option2 job like: ``` { "id": xxx, "os": "openeuler", ... "buildinfo": { "Hash": "SHA512", "Format": 1.0, ... }, "stats": { "buildinfo.Hash": "SHA512", "buildinfo.Format": 1.0, ... } } ``` **advantage: opton1 + option2 ** **disadvantage:** 1. job bigger, --> save DB 2. extract-service does one more thing

1 0

Re: reproducible-build TODO list
by Lu Weitao 09 Feb '22

09 Feb '22

On Tue, Feb 08, 2022 at 08:20:37PM +0800, Wei Jihui wrote: >On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >>## TODO list >>- TODO1.1: script support set env for build env --- compass-ci jihui > >what should i do for this? > user submit rpmbuild.yaml will with some build params like: fs: kernel: reproducible_build_env: hostname: datetime: rpmbuild: new_param: we should make sure these params take effect in testbox --------- Thanks, Lu Weitao - >Thanks, >Jihui > > >> >>- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci >> - based on debian buildinfo >> - TODO1.3: 输出用户指定的息into buildinfo? >> new lib/buildinfo.sh --- IT >> func1() >> func2() >> func3() >> save to $result_root/buildinfo.yaml >> >> modify rpmbuild script to call the above funcs. --- compass-ci wangyong >> >>- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao >> >>- TODO2: 比较构建结果是否相同 --- compass-ci weitao >> compass ci >> >> reuse data API es-find >> compare in client side >> >> 格式和进一步优化: IT >> client: ask use python? >> >>- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT >> >>- TODO4 toolset --- IT/compass-ci >> option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 >> option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT >> option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong >> >> >> >>--------- >>Thanks, >>Lu Weitao >>

1 0

Re: [PATCH lkp-tests 3/3] lkp-tests/*: add new flow for install lkp depends
by Wang Yong 09 Feb '22

09 Feb '22

On Tue, Feb 08, 2022 at 11:28:38AM +0800, Wei Jihui wrote: >On Sun, Jan 30, 2022 at 08:58:23AM +0800, Wang Yong wrote: >>[why] >>install lkp depends before test suite to satisfy lkp-tests >> >>Signed-off-by: Wang Yong <wangyong187(a)huawei.com> >>--- >> lib/job2sh.rb | 2 ++ >> setup/install_depends | 49 +++++++++++++++++++++++++++++++++++-------- >> 2 files changed, 42 insertions(+), 9 deletions(-) >> >>diff --git a/lib/job2sh.rb b/lib/job2sh.rb >>index 4e8bf97fb..32caf2ade 100755 >>--- a/lib/job2sh.rb >>+++ b/lib/job2sh.rb >>@@ -316,6 +316,7 @@ class Job2sh < Job >> @monitors = available_programs(:monitors) >> @setups = available_programs(:setup) >> @programs = available_programs(:workload_elements) >>+ @lkp = {"install_depends"=>{}} > >how about use '@install_depends'? > ok Thanks, Yong

1 0

Re: reproducible-build TODO list
by weijihui 08 Feb '22

08 Feb '22

On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >## TODO list >- TODO1.1: script support set env for build env --- compass-ci jihui what should i do for this? Thanks, Jihui > >- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci > - based on debian buildinfo > - TODO1.3: 输出用户指定的息into buildinfo? > new lib/buildinfo.sh --- IT > func1() > func2() > func3() > save to $result_root/buildinfo.yaml > > modify rpmbuild script to call the above funcs. --- compass-ci wangyong > >- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao > >- TODO2: 比较构建结果是否相同 --- compass-ci weitao > compass ci > > reuse data API es-find > compare in client side > > 格式和进一步优化: IT > client: ask use python? > >- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT > >- TODO4 toolset --- IT/compass-ci > option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 > option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT > option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong > > > >--------- >Thanks, >Lu Weitao >

1 0

Re: [PATCH compass-ci] container/data-api/data-api: fix incorrect comments
by Lu Weitao 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 09:39:14AM +0800, Wei Jihui wrote: >On Mon, Feb 07, 2022 at 11:22:13AM +0800, Lu Weitao wrote: >>Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >>--- >> container/data-api/data-api | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >>diff --git a/container/data-api/data-api b/container/data-api/data-api >>index 2bd6e527..c1c8af8f 100755 >>--- a/container/data-api/data-api >>+++ b/container/data-api/data-api >>@@ -17,7 +17,7 @@ set :environment, :production >> >> # POST >> # eg1: >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es_find -H 'Content-Type: application/json' -d '{ > > >we will define new api name for data_api. > ok --------- Thanks, Lu Weitao >Thanks, >Jihui > > >> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >> # "index": "jobs", >> # "my_token": "16132550-...", >>@@ -25,7 +25,7 @@ set :environment, :production >> # "query_type": "eql" # can be omitted in this scenario >> # }' >> # eg2: >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es_find -H 'Content-Type: application/json' -d '{ >> # "query": "SELECT * FROM jobs WHERE ...", >> # "index": "jobs", >> # "my_token": "16132550-...", >>-- >>2.23.0 >>

1 0

Re: [PATCH v2 compass-ci] data-api: add es search api
by Wei Jihui 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 10:28:21AM +0800, Wu Fengguang wrote: >>+ def self.search(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] || {'query' => {}} >>+ >>+ return "index is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) > >这里的return会在上一级被作为HTTP 200 >但看起来是一个错误情况。是的，抛出错误更好一些。 Thanks, Jihui > >>+ def self.search_by_sql(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] >>+ >>+ return "query table is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) > >同上。 > >Thanks, >Fengguang >

1 0

Re: [PATCH v2 compass-ci] data-api: add es search api
by Wei Jihui 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 10:24:24AM +0800, Wu Fengguang wrote: >>+module EsDataApi >>+ ES_ACCOUNTS = ESClient.new(index: 'accounts') >>+ OPEN_INDEX = Set.new(['jobs']) >>+ REQUIRED_TOKEN_INDEX = Set.new(['jobs']) >>+ ES_QUERY_KEYWORD = Set.new(['term', 'match']) >>+ >>+ > >2 empty lines can be merged. ok. > >>+ def self.search(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] || {'query' => {}} >>+ >>+ return "index is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) >>+ >>+ if REQUIRED_TOKEN_INDEX.include?(index) >>+ cci_authorization = request_body['cci_authorization'] || {} >>+ my_account = cci_authorization['my_account'] >>+ my_token = cci_authorization['my_token'] >>+ raise "user authentication failed" unless verify_user(my_account, my_token) > >加上my_account，把错误信息变得友好些。 ok. user authentication failed，please check my_account and my_token. > >>+ > >函数体有点大。功能型代码块，适合独立拆分出去。 ok. >>+ def self.verify_user(my_account, my_token) >>+ raise "missed my_account" unless my_account >>+ raise "missed my_token" unless my_token >>+ >>+ query = {} >>+ query['my_account'] = my_account if my_account >>+ query['my_token'] = my_token if my_token > >这里的两个if前面判断过了，可省略 > ok Thanks, Jihui >>+ result = ES_ACCOUNTS.multi_field_query(query)['hits']['hits'] >>+ return nil if result.size == 0 >>+ >>+ result[0]['_source']['my_token'] == my_token >>+ end >>+end >>-- >>2.23.0 >>

1 0

Re: [PATCH compass-ci] src/lib/web_backend.rb: set page_num and page_size to a default value.
by Wei Jihui 08 Feb '22

08 Feb '22

On Mon, Feb 07, 2022 at 04:02:52PM +0800, Li Ping wrote: >[why] >if no parameter is transferred, the default value is not set, As a >result, data cannot be obtained. > 我已经撤销了这个patch,合入最新的patch。 Thanks, Jihui >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > src/lib/web_backend.rb | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > >diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb >index f47950e6..571d7c64 100644 >--- a/src/lib/web_backend.rb >+++ b/src/lib/web_backend.rb >@@ -1026,8 +1026,10 @@ end > def get_srpm_info(params) > begin > my_data = MyData.new >+ page_size = params['page_size'] || 10 >+ page_num = params['page_num'] || 0 > info=[] >- srpm_infos = my_data.get_srpm_info(size: params['page_size'].to_i, from: params['page_num'].to_i) >+ srpm_infos = my_data.get_srpm_info(size: page_size, from: page_num) > srpm_infos['hits']['hits'].each do |source| > info << source['_source'] > end >-- >2.23.0 >

1 0

Re: [PATCH lkp-tests 3/3] lkp-tests/*: add new flow for install lkp depends
by Wei Jihui 08 Feb '22

08 Feb '22

On Sun, Jan 30, 2022 at 08:58:23AM +0800, Wang Yong wrote: >[why] >install lkp depends before test suite to satisfy lkp-tests > >Signed-off-by: Wang Yong <wangyong187(a)huawei.com> >--- > lib/job2sh.rb | 2 ++ > setup/install_depends | 49 +++++++++++++++++++++++++++++++++++-------- > 2 files changed, 42 insertions(+), 9 deletions(-) > >diff --git a/lib/job2sh.rb b/lib/job2sh.rb >index 4e8bf97fb..32caf2ade 100755 >--- a/lib/job2sh.rb >+++ b/lib/job2sh.rb >@@ -316,6 +316,7 @@ class Job2sh < Job > @monitors = available_programs(:monitors) > @setups = available_programs(:setup) > @programs = available_programs(:workload_elements) >+ @lkp = {"install_depends"=>{}} how about use '@install_depends'? Thanks, Jihui > > unless job > job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job >@@ -336,6 +337,7 @@ class Job2sh < Job > out_line > out_line "\texport_top_env" > out_line >+ parse_hash [], @lkp > parse_hash [], job > out_line "}\n\n" > @script_lines >diff --git a/setup/install_depends b/setup/install_depends >index b5390cad1..8a6da808e 100755 >--- a/setup/install_depends >+++ b/setup/install_depends >@@ -1,6 +1,7 @@ > #!/bin/sh > > . $LKP_SRC/lib/detect-system.sh >+. $LKP_SRC/lib/install.sh > > detect_system > >@@ -16,8 +17,6 @@ set_ubuntu_debian() > -e "s|http://archive.ubuntu.com|${mirror_addr}|g" \ > -i.bak \ > /etc/apt/sources.list >- >- apt-get install -yqm "${packages}" > } > > set_suse() >@@ -25,8 +24,6 @@ set_suse() > sed -e "s|^baseurl=http://download.opensuse.org|baseurl=${mirror_addr}/opensuse|g" \ > -i.bak \ > /etc/zypp/repos.d/repo-*.repo >- >- zypper -q install -y "${packages}" > } > > set_archlinux() >@@ -36,8 +33,6 @@ set_archlinux() > else > sed -i "1s;^;Server\\ =\\ ${mirror_addr}/archlinux/\$arch/\$repo\\n;" /etc/pacman.d/mirrorlist > fi >- >- pacman -Syq --noconfirm --needed "${packages}" > } > > set_fedora() >@@ -46,8 +41,6 @@ set_fedora() > -e "s|^#baseurl=http://download.example/pub/fedora/linux|baseurl=${mirror_addr}/fedora|g" \ > -i.bak \ > /etc/yum.repos.d/fedora*.repo >- >- dnf install -y -q "${packages}" > } > > set_centos() >@@ -74,7 +67,43 @@ set_centos() > ;; > > esac >- yum install -y -q "${packages}" >+} >+ >+adapt_lkp_depends() >+{ >+ os=$(echo "${_system_name}" | tr '[:upper:]' '[:lower:]') >+ lkp_depends=$(get_dependency_packages $os lkp | xargs) >+} >+ >+get_package_manager() >+{ >+ has_cmd "yum" && installer="yum" >+ has_cmd "dnf" && installer="dnf" && return >+ has_cmd "apt-get" && installer="apt-get" && return >+ has_cmd "pacman" && installer="pacman" && return >+ has_cmd "zypper" && installer="zypper" && return >+} >+ >+install_depends() >+{ >+ packages="$1" >+ >+ get_package_manager >+ case "$installer" in >+ apt-get) >+ export DEBIAN_FRONTEND=noninteractive >+ "$installer" install -yqm ${packages} >+ ;; >+ dnf|yum) >+ "$installer" install -y -q --skip-broken ${packages} >+ ;; >+ pacman) >+ "$installer" -Sy --noconfirm --needed ${packages} >+ ;; >+ zypper) >+ "$installer" -q install -y ${packages} >+ ;; >+ esac > } > > set_local_mirror() >@@ -104,3 +133,5 @@ set_local_mirror() > } > > set_local_mirror >+adapt_lkp_depends >+install_depends "${lkp_depends}" >-- >2.23.0 >

1 0

test mail for mutt
by Dong Bingqi 08 Feb '22

08 Feb '22

test msg

1 0

test mail for mutt
by Li Yuanchao 08 Feb '22

08 Feb '22

test msg

1 0

help
by 小青 07 Feb '22

07 Feb '22

1 0

test msg to lsd 2
by Wang Yong 07 Feb '22

07 Feb '22

test msg

1 0

test mail to lsd
by Wu Zhende 07 Feb '22

07 Feb '22

test mail

1 0

test mail to wjh
by Luan Shengde 07 Feb '22

07 Feb '22

test mail

1 0

test mail to lsd
by Wang Lin 07 Feb '22

07 Feb '22

test msg

1 0

test mail to wl
by Luan Shengde 07 Feb '22

07 Feb '22

test msgg

1 0

test mail
by Wang Lin 07 Feb '22

07 Feb '22

test msg

1 0

reproducible-build TODO list
by Lu Weitao 07 Feb '22

07 Feb '22

## TODO list - TODO1.1: script support set env for build env --- compass-ci jihui - TODO1.2: script record buildinfo file in testbox --- IT/compass-ci - based on debian buildinfo - TODO1.3: 输出用户指定的息into buildinfo? new lib/buildinfo.sh --- IT func1() func2() func3() save to $result_root/buildinfo.yaml modify rpmbuild script to call the above funcs. --- compass-ci wangyong - TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao - TODO2: 比较构建结果是否相同 --- compass-ci weitao compass ci reuse data API es-find compare in client side 格式和进一步优化: IT client: ask use python? - TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT - TODO4 toolset --- IT/compass-ci option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong --------- Thanks, Lu Weitao

1 0

test mail 0207
by Lu Weitao 07 Feb '22

07 Feb '22

test msg 0207

1 0

Re: test mail to cxl
by Wu Fengguang 29 Jan '22

29 Jan '22

Nice to meet you, Shengde! :)

1 0

Re: test mail to lsd
by Wu Fengguang 29 Jan '22

29 Jan '22

Weitao, Got it. :) Thanks, Fengguang

1 0

test mail to lsd
by Lu Weitao 29 Jan '22

29 Jan '22

test mail bbb

1 0

test mail to cxl
by Luan Shengde 29 Jan '22

29 Jan '22

test msg aaa

1 0

test mail to cxl
by Luan Shengde 29 Jan '22

29 Jan '22

test msg

1 0

test mail
by Luan Shengde 29 Jan '22

29 Jan '22

test msg aaa

1 0

test mail 0129-e
by luanshd0525 29 Jan '22

29 Jan '22

test msg

1 0

test mail 0129-a
by Luan Shengde 29 Jan '22

29 Jan '22

test msg

1 0

test mail 0129
by Luan Shengde 29 Jan '22

29 Jan '22

test msg 0129

1 0

test mail for subscription
by luanshd0525 28 Jan '22

28 Jan '22

test msg for subscription

1 0

test mail for mailing list
by 暗夜 28 Jan '22

28 Jan '22

test msg for mailing list

1 0

test for mailing list
by luanshd 28 Jan '22

28 Jan '22

test msg for mailing list

1 0

[PATCH compass-ci] doc: add doc of rootfs
by Wu Fengguang 05 Dec '21

05 Dec '21

Signed-off-by: Yu Chuan <13186087857(a)163.com> --- doc/help/rootfs/README.md | 146 ++++++++++++++++++ .../compass-ci-use-rootfs/cifs-nfs-hw.md | 58 +++++++ .../compass-ci-use-rootfs/cifs-nfs-qemu.md | 108 +++++++++++++ .../rootfs/compass-ci-use-rootfs/cifs-nfs.md | 7 + .../rootfs/compass-ci-use-rootfs/container.md | 86 +++++++++++ .../rootfs/compass-ci-use-rootfs/initramfs.md | 80 ++++++++++ .../rootfs/compass-ci-use-rootfs/local.md | 80 ++++++++++ doc/help/rootfs/how-to-get-rootfs/README.md | 35 +++++ .../auto-generate-cifs-nfs.md | 97 ++++++++++++ .../manual-generate-cifs-nfs-local.md | 58 +++++++ .../manual-generate-initramfs.md | 39 +++++ 11 files changed, 794 insertions(+) create mode 100644 doc/help/rootfs/README.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/container.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/initramfs.md create mode 100644 doc/help/rootfs/compass-ci-use-rootfs/local.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/README.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md create mode 100644 doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md diff --git a/doc/help/rootfs/README.md b/doc/help/rootfs/README.md new file mode 100644 index 000000000000..eac1ea01ee58 --- /dev/null +++ b/doc/help/rootfs/README.md @@ -0,0 +1,146 @@ +[TOC] + +1 概念解释 +========== + +1.1 Compass-CI的rootfs是什么？ +------------------------------ + +Compass-CI的rootfs是指存放在服务器端的，有版本的，可以为执行机在执行指定os,os_arch,os_version的job时提供相同的，干净的执行环境的docker镜像/文件/目录。 + +Compass-CI当前可供测试的执行机类型有：docker, 虚拟机, 物理机。 +- 对docker而言，rootfs就是docker registry里面的指定版本的docker image。 +- 对虚拟机和物理机而言，rootfs就是在指定版本的iso安装完毕后，将其所有文件（运行时文件除外）拷贝出来，归档到/srv/os或/srv/initrd下版本对应的目录/文件。 + +1.2 os_mount是什么？ +-------------------- + +Compass-CI的job执行是通过submit命令来触发的，os_mount是submit时候的一个参数，用来指定本次job执行所需os是通过何种方式提供的。 + +|----------------------------------|-------------------------------------------------------------------------------| +| Compass-CI目前支持的os_mount类型 | 简单说明 | +|----------------------------------|-------------------------------------------------------------------------------| +| container | 对应执行环境为docker | +| initramfs | 内存文件系统，系统所需文件一次性全部加载到内存中 | +| nfs | 内存文件系统，系统所需文件是通过nfs协议mount的，使用到多少文件，加载多少文件 | +| cifs | 内存文件系统，系统所需文件是通过cifs协议mount的，使用到多少文件，加载多少文件 | +| local | 硬盘文件系统，系统所需文件是位于一个指定名称格式的逻辑卷中 | +|----------------------------------|-------------------------------------------------------------------------------| + +1.3 rootfs与os_mount的关系是什么？ +---------------------------------- + +job通过submit命令来提交到compass-ci调度器； + +submit job时候需要指定os_mount，若不指定，默认值如下： +- 若指定本次执行机类型为docker，则os_mount默认为container； +- 若指定本次执行机类型为虚拟机/物理机，则os_mount默认为initramfs； + +rootfs需要在sbumit job之前制作好，否则submit job会得到错误返回。 + +|--------------|-----------------------------------------------------------------| +| os_mount类型 | 对应的rootfs存储位置 | +|--------------|-----------------------------------------------------------------| +| container | Compass-CI集群的docker registry。image格式：${os}:${os_version} | +| initramfs | /srv/initrd/osimage/${os}/${os_arch}/${os_version}/current | +| cifs/nfs | /srv/os/${os}/${os_arch}/${os_version} | +| local | /srv/os/${os}/${os_arch}/${os_version}-iso | +|--------------|-----------------------------------------------------------------| + +注： +local与cifs/nfs现阶段的区别如下： +- cifs/nfs所需的rootfs目前是通过安装iso到虚拟机得到的； +- local所需的rootfs目前是通过安装iso到物理机得到的； +- 理论上： + - 就与手动安装iso到物理机的系统差异而言，安装iso到物理机得到的rootfs，差异是小于安装iso到虚拟机得到的rootfs的； + - 所以，local能用的rootfs，是可以直接用于cifs/nfs的，而且以后如果安装iso到物理机得到rootfs的方式自动化完毕，那么在磁盘的/srv/os下，是只需要保留安装iso到物理机得到的rootfs的，现阶段local需要的-iso后缀可以去掉，cifs/nfs/local，都使用同一套rootfs。 + + +2 使用指南 +========== + +[提交job](../../job/submit/submit-job.zh.md) +[os_mount](../../job/fields/os_mount.md) +如何查看当前compass-ci集群支持的rootfs都有哪些？ # 待补充 + + +3 管理员手册 +============ + +3.1 Compass-CI的rootfs是怎么来的？（原理） +------------------------------------------ + +### 3.1.1 container + +基于指定版本的基础镜像基础，制作docker image，并上传到docker registry中 + +### 3.1.2 nfs/cifs + +- 安装iso到虚拟机的硬盘（qcow2）上； +- 安装完成后，使用硬盘系统再进行一次启动/关机操作； +- 将qcow2中的rootfs拷贝出来； # 可以使用compass-ci/container/qcow2rootfs容器进行拷贝 +- 对拷贝出来的rootfs进行一些post配置，即可用于compass-ci。 + +> 详细post配置见之后的rootfs如何制作部分 + +### 3.1.3 local + +- 安装iso到物理机的硬盘A上； +- 安装完成后，使用硬盘A上的系统再进行一次启动/关机操作； +- 使用非安装iso的硬盘A启动系统（可以使用内存文件系统，也可以使用其他硬盘上的系统），将硬盘A中的rootfs拷贝出来； +- 对拷贝出来的rootfs进行一些post配置，即可用于compass-ci。 + +> 详细post配置见之后的rootfs如何制作部分 + +### 3.1.4 initramfs + +iniramfs所需的rootfs是一个包含了所有系统必须文件的cgz文件。 +所以我们使用cpio将cifs/nfs/local制作出来的rootfs打包制作成cgz文件即可。 +假设此rootfs位于B目录，将B目录整体打包。 + +> 详细打包命令见之后的rootfs如何制作部分 + + +3.2 我已有Compass-CI集群，如何获得rootfs？ +------------------------------------------ + +[3.2 我已有Compass-CI集群，如何获得rootfs？](./how-to-get-rootfs/README.md) + + +3.3 我已有Compass-CI集群，已有rootfs，如何更新？ +------------------------------------------------ + +### 3.3.1 更新os_mount=container的rootfs + +根据docker registry里的基线版本启动一个docker container，进行需要的更新后，覆盖提交到docker registry。 + +### 3.3.2 更新os_mount=cifs/nfs/local的rootfs + +根据/srv/os/${os}/${os_arch}/${os_version}这个软链接指向的基线版本目录复制一个新的版本目录，在新版本目录中进行需要的更新后，将软链接指向新的版本。 + +### 3.3.3 更新os_mount=initramfs的rootfs + +将/srv/initrd/osimage/${os}/${os_arch}/${os_version}/current这个软链接指向的基线版本cgz解压到一个临时目录，在临时版本目录中进行需要的更新后，将临时目录重新打包为新的版本，并将软链接指向新的版本。 + +解压cgz命令参考： +``` +unzip_dir="$(date "+%Y%m%d%H%M")-unzip" + +mkdir -p $unzip_dir && cd $unzip_dir +zcat < $(realpath /srv/initrd/osimage/${os}/${os_arch}/${os_version}/current) | cpio -idmv +``` + +制作cgz命令参考： +- 参考之前的手动制作initramfs rootfs步骤 + + +3.4 Compass-CI内部是如何使用rootfs的？ +-------------------------------------- + +[3.4.1 os_mount=container的rootfs是如何使用起来的](./compass-ci-use-rootfs/container.md) + +[3.4.2 os_mount=cifs/nfs的rootfs是如何使用起来的](./compass-ci-use-rootfs/cifs-nfs.md) + +[3.4.3 os_mount=local的rootfs是如何使用起来的](./compass-ci-use-rootfs/local.md) + +[3.4.4 os_mount=initramfs的rootfs是如何使用起来的](./compass-ci-use-rootfs/initramfs.md) diff --git a/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md new file mode 100644 index 000000000000..9812bf439363 --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-hw.md @@ -0,0 +1,58 @@ +#### 3.4.2.2 物理机使用cifs/nfs的rootfs流程 + +物理机类型的testbox需要设置物理机的首选启动项为PXE，它的运行是从上电启动开始的。 + +所以我们分析rootfs如何使用，从上电开始分析。 + +1. testbox启动并与调度器建立websocket连接，等待job + +- 物理机上电启动，进入到pxe阶段，pxe阶段会向同一网络发送pxe请求； + +- 这时Compass-CI集群提供的dnsmasq服务，会接收到这个pxe请求，返回给物理机一个可用的ipxe驱动下载地址； # 如：/tftpboot/ipxe/bin-arm64-efi/snp.efi + +- 物理机拿到并加载ipxe驱动，加载完成后会再次向同一网络发出ipxe请求； + +- 这时Compass-CI集群提供的dnsmasq服务，会接收到这个ipxe请求，返回给物理机一个tftp ipxe文件下载地址； # 如：boot.ipxe （此处为相对路径，即文件真实路径：/tftpboot/boot.ipxe） + +- 物理机拿到并执行ipxe命令行所组成的boot.ipxe。 + - boot.ipxe内容如下： + ``` + #!ipxe + set scheduler 172.168.131.113 + set port 3000 + + chain http://${scheduler}:${port}/boot.ipxe/mac/${mac:hexhyp} + + exit + ``` + - 所以，到此步骤，物理机testbox已经与调度器建立了连接，等待job。 + + - 服务端（调度器）如果半个小时都没有调度到这个客户端的任务，就会给客户端返回如下返回值: + ``` + chain http://#{ENV["SCHED_HOST"]}:#{ENV["SCHED_PORT"]}/boot.ipxe/mac/${mac:hexhyp}" + ``` + - 物理机接收到此返回值，会继续请求job + + - 服务端（调度器）如果找到了对应的job，会返回给客户端经过组合的返回值。（相当于客户端接收到了job）。 + 返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + +2. testbox（客户端）接收到job，开始准备环境，执行job + +- 客户端解析并执行服务端（调度器）传回来的返回值，开始启动 + + - 服务端传回来的返回值，直接是由ipxe命令组成的字符串，所以，这个字符串会被ipxe直接解析执行 + - 如上面的例子，这个字符串的最后一行是“boot”，所以，物理机此时，会进入Compass-CI自定义化的Linux开机流程。 + +- Compass-CI的自定义化的linux开机流程 + 由于都是os_mount=cifs/nfs，所以，物理机的开机流程与3.4.2.1的“虚拟机使用cifs/nfs的rootfs流程”部分的开机流程一致。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md new file mode 100644 index 000000000000..4195f5ab2a4a --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs-qemu.md @@ -0,0 +1,108 @@ +#### 3.4.2.1 虚拟机使用cifs/nfs的rootfs流程 + +虚拟机类型的testbox是通过调用/c/compass-ci/providers/qemu.sh脚本，同时给其传入各种运行时参数，运行起来的。 + +所以我们分析rootfs如何使用，从这个脚本开始分析。 + +1. testbox启动并与调度器建立websocket连接，等待job + +- /c/compass-ci/providers/qemu.sh + - functions: + main() + - key code: + source "$CCI_SRC/providers/$provider/${template}.sh" + +- /c/compass-ci/providers/qemu/kvm.sh + - functions: + write_logfile() + - key code: + ```bash + url=ws://${SCHED_HOST:-172.17.0.1}:${SCHED_PORT:-3000}/ws/boot.ipxe/mac/${mac} + ipxe_script_path="$(pwd)/${ipxe_script}" + command -v ruby && + ruby -r "${CCI_SRC}/providers/lib/common.rb" -e "ws_boot '$url','$hostname','$index','$ipxe_script_path'" + ``` + - 说明： + - ws_boot()是位于/c/compass-ci/providers/lib/common.rb中的，compass-ci封装的一个方法; + - ws_boot()运行的一侧，属于客户端，它会与服务端（调度器）建立websocket长链接，请求job； + - 服务端（调度器）如果半个小时都没有调度到这个客户端的任务，就会给客户端返回包含“no job now”的返回值； + /c/compass-ci/providers/qemu/kvm.sh中会处理返回值： + - 如果返回值包含“no job now”，那么继续循环请求job； + - 服务端（调度器）如果找到了对应的job，会返回给客户端经过组合的返回值。（相当于客户端接收到了job）。 + 返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + +2. testbox（客户端）接收到job，开始准备环境，执行job + +- /c/compass-ci/providers/qemu/kvm.sh + - functions: + write_logfile() > parse_ipxe_script() > run_qemu() + - key code: + ```bash + ################### + # func: parse_ipxe_script() + # - 解析调度器传回来的返回中的kernel那一行 + # - kernel那一行举例: kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + ################### + + kernel) + kernel=$(basename "$b") + wget --timestamping -nv -a ${log_file} $b + append=$(echo "$c" | sed -r "s/ initrd=[^ ]+//g") + ;; + + ################### + # func: run_qemu() + # - 使用一些参数启动qemu + # - ${append}是从上面解析得来的,请注意上面例子中kernel后面的3个关键字段： + # - overlay + # - initrd=initramfs.lkp-5.3.18-57-default.img + # - root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino + ################### + + "${kvm[@]}" "${arch_option[@]}" --append "${append}" + ``` + - 说明： + - 到此步，我们已经知道了，qemu是通过命令调用起来的，本次job所需的kernel, initrd(s), rootfs均以参数的形式传给了这个命令。 + - 接下来，就到了Compass-CI的自定义化的linux开机启动流程。 + +- Compass-CI的自定义化的linux开机流程 + + - Linux开机流程简要说明： + - 开机启动内核，内核加载initrd(s)中的所有文件到内存； # initrd(s)所有文件组合起来，会是一个文件系统 + - 在内存中依次执行initrd(s)组成的系统中定义的启动项； + - 在执行这些启动项的时候，会根据传入的内核命令行参数（也就是${append}），来找到并启动本次要真正使用的文件系统。 # 这一步骤，依然还是在initrd(s)组成的文件系统中 + - 启动本次真正要使用的文件系统。 # 这就是所谓的“切根” —— 使用的根文件系统，从initrd(s)组成的文件系统，“切”到真正要使用的文件系统 + + - 对cifs/nfs类型的job： + - 它真正要使用的文件系统，就是root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino + + - 我们传入的initrd(s)组成的文件系统，包括了当前这台testbox执行分配给它的job所需要的文件； # 如lkp.cgz，job.cgz中的文件 + - 但是这些文件是在initrd(s)组成的文件系统中的，而不在本次要真正使用的文件系统中； + - 相当于在initrd(s)组成的文件系统（/目录）中，有我们job需要的文件的； + - 而本次要真正使用的文件系统（/sysroot目录）中，没有我们job需要的文件的； # root=cifs://xxxx会通过指定的协议（cifs/nfs）挂载到/sysroot目录 + + - 所以，我们需要把job需要的文件从/目录，拷贝到/sysroot目录。 + - 这个拷贝的动作，是由initrd(s)中的某一个开机启动项来触发的。 + + - 在cifs/nfs类型的job，由于我们传入的内核命令行参数中有overlay这个关键字，而overlay这个关键字，对应的有其开机启动项，拷贝的动作就是在这个开机启动项触发的。 + - overlay对应的开机启动项，在默认的initrd中是不支持的，所以我们每做出一个rootfs，都需要通过/c/compass-ci/container/dracut-initrd这个容器来生成一个initrd。 + - 在本例中，就是initrd=initramfs.lkp-5.3.18-57-default.img + - 而cifs/nfs类型的job，这个拷贝的动作对应的代码，位于：/c/compass-ci/container/dracut-initrd/bin/overlay-lkp.sh + + - 拷贝前：root=cifs://xxxx已经通过指定的协议（cifs/nfs）挂载到/sysroot目录了 + - 拷贝中：job所需要的文件会被拷贝到job真正要使用的文件系统（/sysroot）中 + - 拷贝后：/sysroot是本次要真正使用的文件系统，接下来就是来启动/sysroot中的文件系统。 + + - 在这个系统中，已经有我们执行lkp所定义的任务所需要的一系列文件。其中就包括一个关键的系统服务：lkp-bootstrap.service。 + - 所以，在/sysroot中的文件系统执行它自己的开机启动流程时，就能通过lkp-bootstrap.service，来执行我们定义好的job。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md new file mode 100644 index 000000000000..dbfe10f6e9ae --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/cifs-nfs.md @@ -0,0 +1,7 @@ +### 3.4.2 os_mount=cifs/nfs的rootfs是如何使用起来的 + +os_mount=cifs/nfs的rootfs会被虚拟机/物理机类型的testbox使用，虚拟机与物理机类型的testbox的启动方式不同，所以分开来讲。 + +[3.4.2.1 虚拟机使用cifs/nfs的rootfs流程](./cifs-nfs-qemu.md) + +[3.4.2.2 物理机使用cifs/nfs的rootfs流程](./cifs-nfs-hw.md) diff --git a/doc/help/rootfs/compass-ci-use-rootfs/container.md b/doc/help/rootfs/compass-ci-use-rootfs/container.md new file mode 100644 index 000000000000..31f961f9fdba --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/container.md @@ -0,0 +1,86 @@ +### 3.4.1 os_mount=container的rootfs是如何使用起来的 + +os_mount=container的rootfs会被docker类型的testbox使用，docker类型的testbox是通过调用/c/compass-ci/providers/docker.rb脚本，同时给其传入各种运行时参数，运行起来的。 + +所以我们分析rootfs如何使用，从这个脚本开始分析。 + +1. testbox启动并与调度器建立websocket连接，等待job + +- /c/compass-ci/providers/docker.rb + - key code: + ```ruby + require_relative './docker/docker' + main(ENV['hostname'], ENV['queues'], ENV['uuid'], ENV['index']) + ``` + +- /c/compass-ci/providers/docker/docker.rb + - functions: + main() > parse_response() + - key code: + ```ruby + require_relative '../lib/common' + ...omit... + response = ws_boot(url, hostname, index) + ``` + - 说明： + - ws_boot()是位于/c/compass-ci/providers/lib/common.rb中的，compass-ci封装的一个方法; + - ws_boot()运行的一侧，属于客户端，它会与服务端（调度器）建立websocket长链接，请求job； + - 服务端（调度器）如果半个小时都没有调度到这个客户端的任务，就会给客户端返回包含“no job now”的返回值； + /c/compass-ci/providers/docker/docker.rb中会处理返回值： + - 如果返回值包含“no job now”，那么继续循环请求job； + - 服务端（调度器）如果找到了对应的job，会返回给客户端经过组合的返回值。（相当于客户端接收到了job）。 + 返回值举例： + ``` + {"job_id"=>"crystal.3583794", "docker_image"=>"centos:7", "initrds"=>"[\"http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3583794/job.cgz\",\"http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz\",\"http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd…"]"} + ``` + +2. testbox（客户端）接收到job，开始准备环境，执行job + +- /c/compass-ci/providers/docker/docker.rb + - functions: + parse_response() > start_container() + - key code: + ```ruby + ################### + # hash: 是服务端返回值解析而来的字典 + # - 如上例: "docker_image"=>"centos:7" + ################### + + docker_image = hash['docker_image'] + system "#{ENV['CCI_SRC']}/sbin/docker-pull #{docker_image}" + ``` + +- /c/compass-ci/sbin/docker-pull + - functions: + main() > local_repository() + - key code: + ```bash + docker pull $DOCKER_REGISTRY_HOST:$DOCKER_REGISTRY_PORT/$image_name 2> /dev/null + ``` + - 说明： + - 到此步，Compass-CI集群的docker registry里面的os_mount=container的rootfs（docker iamge），就被pull到执行机本地了。 + +- /c/compass-ci/providers/docker/docker.rb + - functions: + start_container() + - key code: + ```ruby + system( + { 'job_id' => hash['job_id'], + 'hostname' => hostname, + 'docker_image' => docker_image, + 'load_path' => load_path, + 'log_dir' => "#{LOG_DIR}/#{hostname}" }, + ENV['CCI_SRC'] + '/providers/docker/run.sh' + ) + ``` + +- /c/compass-ci/providers/docker/run.sh + - key code: + ```bash + docker run + ...omit... + ${docker_image} + ``` + - 说明： + - 到此步，Compass-CI集群的docker registry里面的os_mount=container的rootfs（docker iamge），就被使用起来了。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/initramfs.md b/doc/help/rootfs/compass-ci-use-rootfs/initramfs.md new file mode 100644 index 000000000000..6b917738565f --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/initramfs.md @@ -0,0 +1,80 @@ +### 3.4.4 os_mount=initramfs的rootfs是如何使用起来的 + +#### 3.4.4.1 虚拟机使用initramfs的rootfs流程 + +虚拟机使用initramfs的rootfs流程与3.4.2.1的“虚拟机使用cifs/nfs的rootfs流程”大致相同，我们这里只说不同点。 + +- 不同点1：调度器的返回值不同 + - cifs/nfs返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - initramfs返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8800/initrd/osimage/openeuler/aarch64/20.03/20210609… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8800/initrd/deps/initramfs/debian/aarch64/sid/run-ip… + initrd http://172.168.131.113:8800/initrd/deps/initramfs/openeuler/aarch64/20.03/l… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3585813/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro rdinit=/sbin/init prompt_ramdisk=0 initrd=20210609.0.cgz initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=run-ipconfig_20201103.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - 分析： + 两种os_mount的不同，核心在于传递给kernel的命令行参数不同，而这个不同，会反映在不同点2：Compass-CI的自定义化的Linux启动流程不同 + +- 不同点2：内核命令行参数不同，导致Compass-CI的自定义化的Linux启动流程不同 + + - os_mount=initramfs的系统，会直接使用在内存中解开的initrd(s)启动的系统，没有“切根”步骤 + - initramfs的系统开机流程简要说明： + - 开机启动内核，内核加载initrd(s)中的所有文件到内存； # initrd(s)所有文件组合起来，会是一个文件系统 + - 在内存中依次执行initrd(s)组成的系统中定义的启动项； # 所以，os_mount=initramfs，它的job的运行环境，是在内存中的一个文件系统。 + + - cifs/nfs与local的内核命令行参数相同项： + - user=lkp + - job=/lkp/scheduled/job.yaml + - ip=dhcp + - initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz + - initrd=job.cgz + - initrd=v2021.09.23.cgz + - initrd=9f87e65401d649095bacdff019d378e6.cgz + - rootfs_disk=/dev/vdb + - crashkernel=auto + - rootovl + - ro + + - cifs/nfs的独立参数项： + - initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img + - 这个initrd文件是我们通过/c/compass-ci/container/dracut-initrd容器制作出来的，它包含一个最小化的文件系统，这个文件系统，是debian的。 + - 但是os_mount=initramfs，由于没有切根的动作，所以它不能有debian系统的文件，它需要的是我们本次job真正要使用的文件系统。 + + - root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino # root设备使用cifsroot/nfsroot，它会调用95cifs/95nfs这个启动项。 + - 说明： + - 从数字上95cifs/95nfs，会先于90overlay-root执行。 + - 所以，在initrd(s)的系统启动阶段，会先执行95cifs/95nfs，挂载远程的cifs/nfs服务端的rootfs到/sysroot； + - 然后执行90overlay-root，会基于/sysroot，再覆盖一层可读写的overlayfs，并将job所需的文件拷贝到这层overlayfs中； + - 然后将这层可读写的overlayfs作为本次要使用的文件系统，并启动它，开始执行job。 + + - initramfs的独立参数项： + - rdinit=/sbin/init # 指定initrd(s)组成的文件系统中，要运行的1号进程 + - prompt_ramdisk=0 # 如上面我们讲的，os_mount=initramfs对应的文件系统是内存文件系统，所以需要这个参数，来让内核支持内存文件系统 + - initrd=run-ipconfig_20201103.cgz # 内存文件系统需要initramfs-tools来支持一些功能，比如支持rd.init，支持ipconfig配置ip等... （详情参考：https://manpages.ubuntu.com/manpages/xenial/man8/initramfs-tools.8.html） + - initrd=20210609.0.cgz # 这个文件是os_mount=initramfs所对应的rootfs，它里面是本次job所指定的${os},${os_arch},${os_version}的所有文件 + + - 说明： + - os_mount=initramfs没有切根动作，initrd(s)组成的系统直接就是job的执行环境。 + +#### 3.4.4.2 物理机使用initramfs的rootfs流程 + +物理机使用initramfs的rootfs流程与[3.4.2.2 物理机使用cifs/nfs的rootfs流程](./cifs-nfs-hw.md)大致相同，不同点见上面的“3.4.4.1 虚拟机使用initramfs的rootfs流程”部分的两个不同点。 diff --git a/doc/help/rootfs/compass-ci-use-rootfs/local.md b/doc/help/rootfs/compass-ci-use-rootfs/local.md new file mode 100644 index 000000000000..3d974ea3c9af --- /dev/null +++ b/doc/help/rootfs/compass-ci-use-rootfs/local.md @@ -0,0 +1,80 @@ +### 3.4.3 os_mount=local的rootfs是如何使用起来的 + +os_mount=local的rootfs会被虚拟机/物理机类型的testbox使用，虚拟机与物理机类型的testbox的启动方式不同，所以分开来讲。 + +#### 3.4.3.1 虚拟机使用local的rootfs流程 + +虚拟机使用local的rootfs流程与3.4.2.1的“虚拟机使用cifs/nfs的rootfs流程”大致相同，我们这里只说不同点。 + +- 不同点1：调度器的返回值不同 + - cifs/nfs返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584856/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-2021-05-18-15-08-52/… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp rootovl ro root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - local返回值举例： + ``` + #!ipxe + + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01… + initrd http://172.168.131.113:8000/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01… + initrd http://172.168.131.113:3000/job_initrd_tmpfs/crystal.3584978/job.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/aarch64/v2021.09.23.cgz + initrd http://172.168.131.113:8800/upload-files/lkp-tests/9f/9f87e65401d649095bacd… + kernel http://172.168.131.113:8000/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01… user=lkp job=/lkp/scheduled/job.yaml ip=dhcp local use_root_partition= save_root_partition= os_version=20.03-iso os_lv_size=10G os_partition= rw root=172.168.131.113:/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01-07 initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz initrd=job.cgz initrd=v2021.09.23.cgz initrd=9f87e65401d649095bacdff019d378e6.cgz rootfs_disk=/dev/vdb crashkernel=auto + boot + ``` + - 分析： + 两种os_mount的不同，核心在于传递给kernel的命令行参数不同，而这个不同，会反映在不同点2：Compass-CI的自定义化的Linux启动流程不同 + +- 不同点2：内核命令行参数不同，导致Compass-CI的自定义化的Linux启动流程不同 + - cifs/nfs与local的内核命令行参数相同项： + - user=lkp + - job=/lkp/scheduled/job.yaml + - ip=dhcp + - initrd=initramfs.lkp-4.19.90-2003.4.0.0036.oe1.aarch64.img + - initrd=modules-4.19.90-2003.4.0.0036.oe1.aarch64.cgz + - initrd=job.cgz + - initrd=v2021.09.23.cgz + - initrd=9f87e65401d649095bacdff019d378e6.cgz + - rootfs_disk=/dev/vdb + - crashkernel=auto + + - cifs/nfs的独立参数项： + - rootovl # 这个参数项会让dracut的90overlay-root，这一开机启动项执行 + - ro # root设备以只读模式挂载，因为有了rootovl，所以我们job对于系统所有的操作，都是在overlay的upper层进行的，并不会影响远程挂载的nfs/cifs服务端的文件 + - root=cifs://172.168.131.113/os/openeuler/aarch64/20.03-2021-05-18-15-08-52,guest,ro,hard,vers=1.0,noacl,nouser_xattr,noserverino # root设备使用cifsroot/nfsroot，它会调用95cifs/95nfs这个启动项。 + - 说明： + - 从数字上95cifs/95nfs，会先于90overlay-root执行。 + - 所以，在initrd(s)的系统启动阶段，会先执行95cifs/95nfs，挂载远程的cifs/nfs服务端的rootfs到/sysroot； + - 然后执行90overlay-root，会基于/sysroot，再覆盖一层可读写的overlayfs，并将job所需的文件拷贝到这层overlayfs中； + - 然后将这层可读写的overlayfs作为本次要使用的文件系统，并启动它，开始执行job。 + + - local的独立参数项： + - local # 这个参数项是我们自定义的，它会让我们给initrd(s)的系统中自定义的的开机启动项90lkp被调用 + - use_root_partition= # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - save_root_partition= # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - os_version=20.03-iso # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - os_lv_size=10G # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - os_partition= # 这个参数项是我们自定义的，它会被90lkp捕获并解析使用 + - rw # root设备以读写模式挂载，这个参数只会在高级用户需要自定义ipxe命令行，且指定的root设备直接是某一块需要可读写的块设备时用到 + - root=172.168.131.113:/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01-07 # 这个是固定的，一方面会执行95nfs，另一方面，在之后的90lkp也会用到它 + - 说明： + - 关于90lkp，请见：/c/compass-ci/container/dracut-initrd/modules.d/90lkp + - 从数字上95nfs，会先于90lkp执行。 + - 所以，在initrd(s)的系统启动阶段，会先执行95nfs，挂载远程的nfs服务端的rootfs到/sysroot（实际上并不会用到它）； + - 然后执行90lkp，会根据传入的内核命令行参数（包括root），来将root对应的rootfs拷贝到指定格式的lv中； # 详见：/c/compass-ci/container/dracut-initrd/bin/set-local-sysroot.sh + - 然后将这个lv作为本次要使用的文件系统，并启动它，开始执行job。 + + +#### 3.4.3.2 物理机使用local的rootfs流程 + +物理机使用local的rootfs流程与[3.4.2.2 物理机使用cifs/nfs的rootfs流程](./cifs-nfs-hw.md)大致相同，不同点见上面的“3.4.3.1 虚拟机使用local的rootfs流程”部分的两个不同点。 + diff --git a/doc/help/rootfs/how-to-get-rootfs/README.md b/doc/help/rootfs/how-to-get-rootfs/README.md new file mode 100644 index 000000000000..74673df0a9b0 --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/README.md @@ -0,0 +1,35 @@ +### 3.2.1 从compass-ci官方下载 + +待补充 + +### 3.2.2 通过自动化工具制作rootfs + +#### 3.2.2.1 自动化制作os_mount=container的rootfs + +待补充 + +#### 3.2.2.2 自动化制作os_mount=cifs/nfs的rootfs + +[3.2.2.2 自动化制作os_mount=cifs/nfs的rootfs](./auto-generate-cifs-nfs.md) + +#### 3.2.2.3 自动化制作os_mount=local的rootfs + +待补充 + +#### 3.2.2.4 自动化制作os_mount=initramfs的rootfs + +待补充 + +### 3.2.3 通过手动方式制作rootfs + +#### 3.2.3.1 手动制作os_mount=container的rootfs + +待补充 + +#### 3.2.3.2 手动制作os_mount=cifs/nfs/local的rootfs + +[3.2.3.2 手动制作os_mount=cifs/nfs/local的rootfs](./manual-generate-cifs-nfs-local.md) + +#### 3.2.3.3 手动制作os_mount=initramfs的rootfs + +[3.2.3.3 手动制作os_mount=initramfs的rootfs](./manual-generate-initramfs.md) diff --git a/doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md b/doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md new file mode 100644 index 000000000000..8a848f947cd0 --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/auto-generate-cifs-nfs.md @@ -0,0 +1,97 @@ +#### 3.2.2.2 自动化制作os_mount=cifs/nfs的rootfs + +**说明：由于自动化天然有其有限性，故此种方式有其支持的[iso范围](https://gitee.com/ycvayne/iso2qcow2/tree/master/conf/ks)，如果需要制作的iso版本不在支持的iso范围中，请参考下面的手动安装iso制作rootfs部分** + +在本地compass-ci集群中，提交iso2rootfs.yaml即可 + +``` +zhangsan@localhost ~% cat iso2rootfs-21.09.yaml +suite: iso2rootfs +category: benchmark +iso2rootfs: + ################# + # iso related fields to be used to generate rootfs + ################# + + iso_os: openeuler + iso_arch: aarch64 + iso_version: 21.09 + + ################# + # place the result rootfs related fields + ################# + + # 1. Result rootfs will be placed in the following location on the + # remote file server: + # - {remote_file_server}/{rootfs_path}/{iso_os}/{iso_arch}/ + # 2. Remote file server protocols current supported: + # - nfs + # - cifs + rootfs_protocol: cifs + rootfs_server: 1.1.1.1 + rootfs_path: os-rw + rootfs_mount_param: guest,vers=1.0,noacl,nouser_xattr + + initramfs_protocol: cifs + initramfs_server: 1.1.1.1 + initramfs_path: initrd/osimage + initramfs_mount_param: port=446,guest,vers=1.0,noacl,nouser_xattr + + ################# + # config rootfs related fields + ################# + + # you can config add some configurations of the result rootfs. + # supported fields: + # - dns: will config /etc/resolv.conf. + # - no_selinux: will disable selinux in /etc/selinux/config. + # - no_fstab: will comment all line in /etc/fstab. + # - enable_repos: will enable all repo file of result rootfs. + config_rootfs: no_selinux, no_fstab + + ## install pkgs for result rootfs + ## - example: vim, git, xxx + rootfs_install_pkgs: + + ################# + # iso srouce related fields + ################# + + # iso url which you want generate rootfs from + # - demo: + # iso_url: http://1.1.1.1/openEuler-20.03-LTS-aarch64-LTS/openEuler-20.03-LTS-aarch64-… + + iso_url: http://1.1.1.1:8000/os/install/iso/openeuler/aarch64/21.09/openEuler-21.09-… + + # dailybuild_iso_url_file: + # - The `dailybuild iso url file` content is the url of a iso. + # - The iso checksum file also exists on the network, and the checksum file path is "{dailybuild_iso_url}.check256sum" + # - if your have this field, the above `iso_url` field will be useless. + # - demo: + # dailybuild_iso_url_file: http://1.1.1.1/dailybuilds/openEuler-20.03-LTS-aarch64-LTS/release_iso + # root@localhost ~% curl http://1.1.1.1/dailybuilds/openEuler-20.03-LTS-aarch64-LTS/release_iso + # http://1.1.1.1//dailybuilds/openEuler-20.03-LTS-aarch64-LTS/1970-01-01-00-0… + dailybuild_iso_url_file: + + ################# + # submit test yaml related fields + ################# + + # submit target tests yaml + ## 1. You can add as many jobs as you like. + ## 2. The following three fields is required for every test job. + test1_yaml: iperf.yaml + test1_os_mount: initramfs + test1_testbox: vm-2p16g + ## the following three fields is required when your job.yaml and + ## script for this test are from the internet. + test1_git_url: + test1_git_yaml: + test1_git_script: + +secrets: + MY_EMAIL: XXXXXXXXXXX(a)163.com + MY_NAME: zhangsan + MY_TOKEN: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX +zhangsan@localhost ~% #submit -m ./iso2rootfs-21.09.yaml testbox=taishan200-2280-2s64p-128g--a108 +``` diff --git a/doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md b/doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md new file mode 100644 index 000000000000..97160f70eb9c --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/manual-generate-cifs-nfs-local.md @@ -0,0 +1,58 @@ +#### 3.2.3.2 手动制作os_mount=cifs/nfs/local的rootfs + +**说明：从本质上来讲，cifs/nfs/local所需的rootfs是可以共用的，所以，如果你制作出了local类型的rootfs，那么cifs/nfs均可以使用此rootfs。** + +1. 手动安装iso到物理机A # 此步骤无说明，请自行百度 + - 硬盘可选任意一块硬盘A； + - 安装完毕，一般会提示“按Enter键重启系统”。 +2. firstboot + - 在1步骤的“按Enter键重启系统”，启动系统后，不要做任何操作，直接reboot再次重启系统 +3. 将物理机A上的硬盘A中的系统拷贝到指定目录。 # 如：/srv/os/${os}/${os_arch}/${os_version}-20211129000000 + - 使用物理机A上的非A硬盘中的系统启动物理机A + - 使用位于物理机A上的其他硬盘的系统启动物理机A + - 可以给物理机A提交一个内存文件系统的job（此job不限制os,os_arch,os_version） + - 在物理机A上，拷贝硬盘A中的系统到指定目录。 + - 通过nfs/cifs协议挂载你的Compass-CI集群主节点的/srv/os/${os}/${os_arch}/${os_version}-20211129000000至/mnt1； # rw读写模式 + - 挂在硬盘A至/mnt2 + - rsync -a /mnt2/. /mnt + - 释放物理机A +4. 对指定目录的rootfs进行一些处理 + - 解压内核并生成${rootfs}/boot/vmlinuz + - 解压内核(ipxe需要)：如果你的内核格式中有gzip字样，请参考下面的解决文档解决 + 参考： + - file: https://gitee.com/wu_fengguang/compass-ci/blob/master/container/qcow2rootfs… + - function: unzip_vmlinuz + - 生成vlinuz软链接（compass-ci代码逻辑需要）： + ``` + root@z9 /srv/os/openeuler/aarch64/20.03-iso# ll boot | grep vmlinuz + -rwxr-xr-x 1 root root 7.1M 2021-06-29 20:56 vmlinuz-0-rescue-8f22ff33e906472498de4a5b3fc087ef + -rw-rw-r-- 1 root root 20M 2021-06-29 21:10 vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 + lrwxrwxrwx 1 root root 43 2021-06-29 21:10 vmlinuz -> ./vmlinuz-4.19.90-2003.4.0.0036.oe1.aarch64 + ``` + - 生成${rootfs}/boot/modules.cgz（compass-ci代码逻辑需要） + 参考： + - file: https://gitee.com/wu_fengguang/compass-ci/blob/master/container/qcow2rootfs… + - function: create_get_modules + - 生成${rootfs}/initrd.lkp（compass-ci代码逻辑需要） + 参考： + - file: https://gitee.com/wu_fengguang/compass-ci/blob/master/container/qcow2rootfs… + - function: create_get_initrd + - 注释掉${rootfs}/etc/fstab的启动项（compass-ci代码逻辑需要） + 参考： + - file: https://gitee.com/wu_fengguang/lkp-tests/blob/master/tests/iso2rootfs + - function: disable_fstab + - 关闭${rootfs}的selinux + 参考： + - file: https://gitee.com/wu_fengguang/lkp-tests/blob/master/tests/iso2rootfs + - function: disable_selinux + - 创建真实版本的软链接 + ``` + root@z9 /srv/os/openeuler/aarch64# ll -d 20.03-iso* + dr-xr-xr-x 1 root root 190 2021-08-24 09:09 20.03-iso-2021-08-24-18-01-07 + dr-xr-xr-x 1 root root 190 2021-08-24 09:09 20.03-iso-2021-08-24-09-07-18 + lrwxrwxrwx 1 root root 29 2021-08-24 18:16 20.03-iso -> 20.03-iso-2021-08-24-18-01-07 + ``` +5. 使用制作出来的rootfs提交job + ``` + root@localhost ~% submit -m -c borrow-1d.yaml testbox=vm-2p16g os=openeuler os_version=20.03-iso os_mount=cifs + ``` diff --git a/doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md b/doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md new file mode 100644 index 000000000000..d7e5553dd165 --- /dev/null +++ b/doc/help/rootfs/how-to-get-rootfs/manual-generate-initramfs.md @@ -0,0 +1,39 @@ +#### 3.2.3.3 手动制作os_mount=initramfs的rootfs + +1. 将制作出来的相同${os},${os_arch},${os_version}的os_mount=cifs/nfs/local的rootfs拷贝一份到临时目录。 + ```bash + mkdir /srv/os/openeuler/aarch64/20.03-iso-tmp + rsync -a /srv/os/openeuler/aarch64/20.03-iso-2021-08-24-18-01-07/. /srv/os/openeuler/aarch64/20.03-iso-tmp/ + ``` +2. 进入临时目录，删掉一些不需要加入到initamfs系统的文件 + + **说明：也可以不删除这些文件，如果你知道你需要它们** + ```bash + cd /srv/os/openeuler/aarch64/20.03-iso-tmp/ + rm -rf lib/modules/* + rm -rf boot/vmlinuz* + rm -rf boot/initr* + ``` + + 3. 生成initramfs的rootfs + ```bash + ############ + # 为什么要去掉${os_version}的-iso后缀： + # - 目前，只有os_mount=local对应的rootfs需要-iso后缀，initramfs不需要 + ############ + + initramfs_cgz=/srv/initrd/osimage/$os/$os_arch/${os_version%-iso}/$(date +"%Y%m%d").0.cgz + mkdir -p $(dirname $initramfs_cgz) + cd /srv/os/openeuler/aarch64/20.03-iso-tmp/ + find . |cpio -o -Hnewc | gzip -9 > $initramfs_cgz + + cd $(dirname $initramfs_cgz) + ln -sf $(basename $initramfs_cgz) current + + ############ + # 如何获得../../../../deps/nfs/debian/aarch64/sid/run-ipconfig.cgz： + # - 从Compass-CI官网下载：https://api.compass-ci.openeuler.org:20008/initrd/deps/nfs/debian/aarch64/sid/ + ############ + + ln -sf ../../../../deps/nfs/debian/aarch64/sid/run-ipconfig.cgz run-ipconfig.cgz + ``` -- 2.23.0

1 0

test
by FengGuang Wu 05 Dec '21

05 Dec '21

1 0

test
by Wu Fengguang 05 Dec '21

05 Dec '21

1 0

Re: [PATCH lkp-tests] tests/rpmbuild: use HTTPS instead
by Liu Shaofei 09 Oct '21

09 Oct '21

On Sat, Oct 09, 2021 at 11:36:27AM +0800, Wang Lin wrote: >change to use https and domain name > >Signed-off-by: Wang Lin <wanglin199(a)huawei.com> >--- > tests/rpmbuild | 8 +++++++- > 1 file changed, 7 insertions(+), 1 deletion(-) > >diff --git a/tests/rpmbuild b/tests/rpmbuild >index 0f5244a1b..9b13591d2 100755 >--- a/tests/rpmbuild >+++ b/tests/rpmbuild >@@ -206,7 +206,13 @@ localinstall_dependent_package() > yum install -y wget > mkdir /tmp/rpm > cd /tmp/rpm >- wget -c -r -np -nd "http://${LKP_SERVER}:${SRV_HTTP_RPM_PORT:-20012}$depend_rpm_dest" >+ local protocol >+ if [ ${#SRV_HTTP_PROTOCOL} -ne 0 ] ; then >+ protocol=$SRV_HTTP_PROTOCOL >+ else >+ protocol="https" >+ fi >+ wget -c -r -np -nd "$protocol://${SCHED_HOST}:${SRV_HTTP_RPM_PORT:-20012}$depend_rpm_dest" perhaps you can write like this: wget -c -r -np -nd "${SRV_HTTP_PROTOCOL:-https}://${SCHED_HOST}:${SRV_HTTP_RPM_PORT:-20012}$depend_rpm_dest" Thanks, Liushaofei > yum localinstall -y *.rpm > } > show_rpm_files() >-- >2.23.0 >

1 0

openEuler Summit 2021 诚挚的邀请您提交演讲议题、成为共建单位、贡献展示方案
by Marketing openEuler 29 Sep '21

29 Sep '21

openEuler Summit 是由欧拉开源社区举办的开发者峰会。全新升级后的首场峰会将于 11 月10日在北京举行。 openEuler 从服务器操作系统，升级为数字基础设施的操作系统，支持 IT、CT、OT 等数字基础设施全场景，覆盖服务器、云、边、嵌入式等各种形态的需求。伴随 openEuler 21.09 的发布，openEuler 已经包含了服务器、云原生、边缘计算和嵌入式的四大应用场景。 openEuler 通过开源开放，不断探索科技创新的边界。开发者、用户、社区贡献者、软件爱好者在 openEuler Summit 汇聚，共同解读 openEuler 的最新版，探讨未来的技术路线，让技术、生态、商业在这里产生奇妙的化学反应。 *本次峰会旨在推动 openEuler 在多样性计算、云计算、边缘计算、嵌入式等技术方向的持续探索和创新。* 开源是一种态度、分享是一种精神。Call for Speaker、Call for Sponsor、Call for Demo 现已全面开放报名。大会报名通道同步开启。 *我们诚挚的邀请您提交演讲议题、发表联合演讲、成为共建单位、贡献展示方案、参与社区建设。* - *大会官网：*https://www.openeuler.org/zh/interaction/summit-list/summit2021/ - *Call for Speaker：*https://shimowendang.com/forms/X6X9jj9KPcdQwVr8/fill - *Call for Sponsor：*https://shimowendang.com/forms/k76zTLKvumYwRdsP/fill - *Call for Demo：*https://shimowendang.com/forms/LjHs8JlsLSsW92kl/fill

1 0

Re: [PATCH compass-ci 2/2] modify url about srv-http
by Liu Shaofei 25 Aug '21

25 Aug '21

On Wed, Aug 25, 2021 at 04:55:29PM +0800, Lu Weitao wrote: >[Why] >compass-ci/container/srv-http may change the communication protocol: > http => https >it's depends on whether there are ssl certificates on the server >we need convert all 'http://compass-ci.api.openeuler.org.*' to >'https://compass-ci.api.openeuler.org.*' > >[How] >scope: gg http.*SRV_HTTP > >lib/mail_bisect_result.rb:44: prefix_srv_result = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" >lib/mail_bisect_result.rb:45: prefix_srv_git = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" >sbin/download-rootfs:38: wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" >sbin/download-rootfs:48: wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage/…" >sbin/download-rootfs:49: wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" >sparrow/0-package/os/openEuler:53: wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >sparrow/0-package/os/openEuler:54: wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > lib/constants.rb | 3 +++ > lib/mail_bisect_result.rb | 4 ++-- > sbin/download-rootfs | 6 +++--- > sparrow/0-package/os/openEuler | 4 ++-- > sparrow/local/compare-results.md | 2 +- > 5 files changed, 11 insertions(+), 8 deletions(-) > >diff --git a/lib/constants.rb b/lib/constants.rb >index d85ca5c3..38e1ec92 100644 >--- a/lib/constants.rb >+++ b/lib/constants.rb >@@ -70,6 +70,9 @@ SRV_HTTP_GIT_PORT ||= config['SRV_HTTP_GIT_PORT'] || ENV['SRV_HTTP_GIT_PORT'] || > > SRV_HTTP_DOMAIN ||= config['SRV_HTTP_DOMAIN'] || ENV['SRV_HTTP_DOMAIN'] || 'api.compass-ci.openeuler.org' > >+SRV_HTTP_PROTOCOL ||= >+ File.exist?('/etc/ssl/certs/web-backend.key') && File.exist?('/etc/ssl/certs/web-backend.crt') ? 'https' : 'http' >+ > ASSISTANT_HOST ||= config['ASSISTANT_HOST'] || ENV['ASSISTANT_HOST'] || '172.17.0.1' > ASSISTANT_PORT ||= config['ASSISTANT_PORT'] || ENV['ASSISTANT_PORT'] || 8101 > >diff --git a/lib/mail_bisect_result.rb b/lib/mail_bisect_result.rb >index 855fcdc9..cff3137b 100644 >--- a/lib/mail_bisect_result.rb >+++ b/lib/mail_bisect_result.rb >@@ -41,8 +41,8 @@ class MailBisectResult > > def compose_mail > subject = "[Compass-CI][#{(a)repo.split('/')[1]}] #{@commit_id[0..9]} #{(a)bisect_error[0].split("\n")[0]}" >- prefix_srv_result = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" >- prefix_srv_git = "http://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" >+ prefix_srv_result = "#{SRV_HTTP_PROTOCOL}://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_RESULT_PORT}" >+ prefix_srv_git = "#{SRV_HTTP_PROTOCOL}://#{SRV_HTTP_DOMAIN}:#{SRV_HTTP_GIT_PORT}" > pkgbuild_repo_url = "we build project with this script:\n#{prefix_srv_git}/git/#{@pkgbuild_repo}/PKGBUILD\n" > first_bad_commit_job_url = "first bad commit result :\n#{prefix_srv_result}#{@first_bad_commit_result_root}/build-pkg\n" > >diff --git a/sbin/download-rootfs b/sbin/download-rootfs >index 0d0d1c7b..9f1d0070 100755 >--- a/sbin/download-rootfs >+++ b/sbin/download-rootfs >@@ -35,7 +35,7 @@ wget_download() > download_os_rootfs() > { > cd $rootfs_dir && { >- wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch}…" >+ wget_download "https://api.compass-ci.openeuler.org:${SRV_HTTP_OS_PORT}/os/${os}/${os_arch…" > > gzip -dc ${os_version}.cgz | cpio -id > rm -rf ${os_version}.cgz >@@ -45,8 +45,8 @@ download_os_rootfs() > download_osimage() > { > cd $osimage_dir && { >- wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage/…" >- wget_download "http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nfs…" >+ wget_download "https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/osimage…" >+ wget_download "https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT}/initrd/deps/nf…" ^-----^-----^-------^--- need to change ${os_mount}/${os}/${os_version} ? Thanks, Liushaofei > } > } > >diff --git a/sparrow/0-package/os/openEuler b/sparrow/0-package/os/openEuler >index 1c61e2c0..399e5e47 100755 >--- a/sparrow/0-package/os/openEuler >+++ b/sparrow/0-package/os/openEuler >@@ -50,8 +50,8 @@ public_network_ok && { > > # lib64lockfile1 from http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… > # lockfile-progs from http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >- wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >- wget_rpm http://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/d… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >+ wget_rpm https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… >+ wget_rpm https://api.compass-ci.openeuler.org:${SRV_HTTP_INITRD_PORT:-20008}/initrd/… || wget_rpm http://abf-downloads.openmandriva.org/rock/repository/$(arch)/main/release/… > > rpm -i --force lib64lockfile1-1.16-1-omv4002.$(arch).rpm lockfile-progs-0.1.18-5-omv4002.$(arch).rpm > rm -rf *.rpm >diff --git a/sparrow/local/compare-results.md b/sparrow/local/compare-results.md >index e555f32e..d82b0165 100644 >--- a/sparrow/local/compare-results.md >+++ b/sparrow/local/compare-results.md >@@ -6,7 +6,7 @@ > submit -m cci-depends.yaml cci-depends.benchmark=iperf > ``` > >-2. 编辑os-matrix-dc.yaml，指定使用centos：7和 [openEuler：20.03-pre](http://api.compass-ci.openeuler.org:20008/initrd/dockerimage/openeuler-pre.tar)（由于dockerhub没有openeuler20.03的镜像，可以下载该链接并使用docker load -i openeuler-pre.tar加载到本地），如需要使用其他os，参考如下文件格式修改即可。 >+2. 编辑os-matrix-dc.yaml，指定使用centos：7和 [openEuler：20.03-pre](https://api.compass-ci.openeuler.org:20008/initrd/dockerimage/openeuler-pre.tar)（由于dockerhub没有openeuler20.03的镜像，可以下载该链接并使用docker load -i openeuler-pre.tar加载到本地），如需要使用其他os，参考如下文件格式修改即可。 > ``` > cat > /c/lkp-tests/jobs/os-matrix-dc.yaml << EOF > os_arch: aarch64 >-- >2.23.0 >

1 0

Re: [PATCH compass-ci 1/2] container/srv-http: support https protocol
by Liu Shaofei 25 Aug '21

25 Aug '21

On Wed, Aug 25, 2021 at 04:55:28PM +0800, Lu Weitao wrote: >[Why] >Some clients(chrome|firefox bowser) can't access srv-http like: > http://api.compass-ci.openeuler.org/pub >because the url will be change as:(it's default behavior of bowser) > https://api.compass-ci.openeuler.org/pub > >so srv-http need support https access > >[How] >with start container/srv-http: > > check current server exists?: > /etc/ssl/certs/web-backend.crt > /etc/ssl/certs/web-backend.key > | | > v v > yes no > config nginx.conf like: config /nginx.conf like: > ... ... > listen $port ssl: listen $port; > ... > | > v > start nginx in container > >Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >--- > container/srv-http/docker_run.sh | 9 +++++++++ > container/srv-http/root/sbin/entrypoint.sh | 13 ++++++++++++- > container/srv-http/start-cci | 2 ++ > container/srv-http/start-git | 2 ++ > container/srv-http/start-initrd | 2 ++ > container/srv-http/start-os | 2 ++ > container/srv-http/start-pub | 2 ++ > container/srv-http/start-result | 2 ++ > container/srv-http/start-rpm | 2 ++ > 9 files changed, 35 insertions(+), 1 deletion(-) > create mode 100644 container/srv-http/docker_run.sh > >diff --git a/container/srv-http/docker_run.sh b/container/srv-http/docker_run.sh >new file mode 100644 >index 00000000..fcd9083d >--- /dev/null >+++ b/container/srv-http/docker_run.sh >@@ -0,0 +1,9 @@ >+#!/bin/bash >+# SPDX-License-Identifier: MulanPSL-2.0+ >+# Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. >+ >+mount_ssl(){ >+ if [ -f "/etc/ssl/certs/web-backend.key" ] && [ -f "/etc/ssl/certs/web-backend.crt" ]; then >+ echo "-v /etc/ssl/certs:/opt/cert" >+ fi defined var: ssl_path="/etc/ssl/certs/" , avoid long contents; >+} >diff --git a/container/srv-http/root/sbin/entrypoint.sh b/container/srv-http/root/sbin/entrypoint.sh >index ced6eda5..d0d354c2 100755 >--- a/container/srv-http/root/sbin/entrypoint.sh >+++ b/container/srv-http/root/sbin/entrypoint.sh >@@ -1,6 +1,17 @@ > #!/bin/sh > >-sed -i "s/listen 11300;/listen $LISTEN_PORT;/g" /etc/nginx/conf.d/default.conf >+ssl_conf="\ ssl_certificate /opt/cert/web-backend.crt;\n\ >+ ssl_certificate_key /opt/cert/web-backend.key;\n\ >+ ssl_session_timeout 5m;\n\ >+ ssl_ciphers BCDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;\n\ >+ ssl_prefer_server_ciphers on;\n" >+ >+if [ -f "/opt/cert/web-backend.key" ] && [ -f "/opt/cert/web-backend.crt" ]; then >+ sed -i "s/listen 11300;/listen $LISTEN_PORT ssl;/g" /etc/nginx/conf.d/default.conf >+ sed -i "/server_name/a $ssl_conf" /etc/nginx/conf.d/default.conf >+else >+ sed -i "s/listen 11300;/listen $LISTEN_PORT;/g" /etc/nginx/conf.d/default.conf >+fi perhaps modify like this: listen="$LISTEN_PORT" if [ -f "/opt/cert/web-backend.key" ] && [ -f "/opt/cert/web-backend.crt" ]; then listen="$listen ssl" sed -i "/server_name/a $ssl_conf" /etc/nginx/conf.d/default.conf fi sed -i "s/listen 11300;/listen $listen;/g" /etc/nginx/conf.d/default.conf Thanks, Liushaofei > nginx > /usr/sbin/php-fpm7 >diff --git a/container/srv-http/start-cci b/container/srv-http/start-cci >index aa539d20..1f160f44 100755 >--- a/container/srv-http/start-cci >+++ b/container/srv-http/start-cci >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-cci > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20011:20011 > -v /srv/cci/libvirt-xml:/srv/cci/libvirt-xml:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-git b/container/srv-http/start-git >index 0b0ff477..5851af6f 100755 >--- a/container/srv-http/start-git >+++ b/container/srv-http/start-git >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-git > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20010:20010 > -v /srv/git/archlinux:/srv/git/archlinux:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-initrd b/container/srv-http/start-initrd >index 41958da2..c8e7eded 100755 >--- a/container/srv-http/start-initrd >+++ b/container/srv-http/start-initrd >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-initrd > > cmd=( >@@ -17,6 +18,7 @@ cmd=( > -v /srv/initrd/osimage:/srv/initrd/osimage:ro > -v /srv/initrd/deps:/srv/initrd/deps:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-os b/container/srv-http/start-os >index 5b63902e..c2012397 100755 >--- a/container/srv-http/start-os >+++ b/container/srv-http/start-os >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-os > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20009:20009 > -v /srv/os:/srv/os:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-pub b/container/srv-http/start-pub >index 6dfd5a44..2224737e 100755 >--- a/container/srv-http/start-pub >+++ b/container/srv-http/start-pub >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-pub > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20006:20006 > -v /srv/pub:/srv/pub:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-result b/container/srv-http/start-result >index 32bdf7df..c9e0eac2 100755 >--- a/container/srv-http/start-result >+++ b/container/srv-http/start-result >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-result > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20007:20007 > -v /srv/result:/srv/result:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >diff --git a/container/srv-http/start-rpm b/container/srv-http/start-rpm >index 2d38298c..62fb1385 100755 >--- a/container/srv-http/start-rpm >+++ b/container/srv-http/start-rpm >@@ -3,6 +3,7 @@ > # Copyright (c) 2020 Huawei Technologies Co., Ltd. All rights reserved. > > . $CCI_SRC/container/defconfig.sh >+. ./docker_run.sh > docker_rm srv-http-rpm > > cmd=( >@@ -13,6 +14,7 @@ cmd=( > -p 20012:20012 > -v /srv/rpm/pub:/srv/rpm/pub:ro > -v /etc/localtime:/etc/localtime:ro >+ $(mount_ssl) > -d > srv-http > ) >-- >2.23.0 >

1 0

openEuler 任务打榜赛6月21日-6月28日周榜单来啦
by Marketing openEuler 29 Jun '21

29 Jun '21

openEuler 任务打榜赛6月21日-6月28日周榜单来啦，快来看看有没有你的名字吧。这里提醒已领取赛题但是还没有加群的小伙伴赶紧联系小助手加群呀。好啦~，下面让我们看一下有哪些开发者上榜吧！ gitee用户账号积分值解决issue/pr数量 yaozc701 20 4 dada5208 16 5 shuyao13 14 4 houlibao 12 3 liuerwuliu 12 3 hw999666 8 2 gc7500 8 2 weidongkl 6 1 shao1112 6 2 wss1235 2 1 恭喜以上开发者，请联系小助手领奖。没有上榜的小伙伴不要灰心，请再接再历。周榜、月榜前十名均有奖励，期待在榜单上早点见到你。添加小助手微信（SoftCorePro）回复“*任务打榜赛*”进入交流群 *了解更多* - 官网：https://openeuler.org/zh/interaction/hdc/2021-developer.html - 我要报名： https://competition.huaweicloud.com/information/1000041420/introduction - 我要做题： https://gitee.com/organizations/src-openeuler/issues?assignee_id=&author_id…

1 0

快来看你是否上榜~！openEuler 任务打榜赛5月榜单出炉
by Marketing openEuler 24 Jun '21

24 Jun '21

经过一个月的角逐，openEuler 任务打榜赛5月榜单来啦。在这一个月的时间里，开发者参加任务打榜赛的热情高涨：总计有*38*人领取*118*道赛题，赛道交流群成员*70*个，这里 *提醒领取赛题但是还没有加群的小伙伴赶紧联系小助手加群*呀。好啦~，下面让我们看一下5月份有哪些开发者上榜吧！ gitee用户账号 gitee邮箱积分值解决issue/pr数量 li-kaiyuan66666 likaiyuan7(a)huawei.com 6 1 yanxiaobing2020 yanxiaobing(a)huawei.com 2 1 zhu-yuncheng zhuyuncheng(a)huawei.com 6 1 polite2anyone zhangyao05(a)outlook.com 10 2 yaozc701 yaozc7(a)chinaunicom.cn 18 2 恭喜以上开发者，请中奖的开发者联系小助手*领奖*。没有上榜的小伙伴不要灰心请再接再历，周榜、月榜前十名均有奖励，期待在榜单上早点见到你。让我们共同期待6月的榜单。 [image: image.png] 添加小助手微信回复“*任务打榜赛*”进入交流群 *了解更多* - 官网：https://openeuler.org/zh/interaction/hdc/2021-developer.html - 我要报名： https://competition.huaweicloud.com/information/1000041420/introduction - 我要做题： https://gitee.com/organizations/src-openeuler/issues?assignee_id=&author_id…

1 1

完成任务，获得荣誉，openEuler 社区的这项比赛有点逆天！
by Marketing openEuler 27 May '21

27 May '21

先说比赛规则独立开发者参赛 *做题得分* *每周结算* 这是任务打榜赛不满足？来看一下*特性命题赛*？独立开发者组队参赛区域赛、半决赛、总决赛过五关斩六将谁可以独占鳌头？你将是*社区贡献者* *你的 ID* 会被铭刻在社区还有*“一”点奖金* [image: WechatIMG16.jpeg]

1 0

Fwd: openEuler Developer Day 启动大会招募环节，报名通道同步开启！
by Marketing openEuler 17 May '21

17 May '21

openEuler 社区的开发者：您好！ openEuler Developer Day 2021 将于*6**月10日*在北京举行。本次大会是由 openEuler 社区发起，主要面向签署了 CLA的核心开发者、南北向生态厂商以及核心用户的一场技术盛会，旨在推动 openEuler 社区在多样性计算、云原生全栈、全场景协同等技术方向的持续探索和创新。 *开放招募* openEuler Developer Day 正式开放招募环节：*Call for Speaker* <https://shimo.im/forms/KgXxt8JCYKYtVXHx/fill>、*Call for Sponsor* <https://shimo.im/forms/vwCC68q3pQvvR3Yc/fill>、*Call for SIG* <https://shimo.im/forms/x6vXyW8QHCGK9QJD/fill>、*Call for Demo* <https://shimo.im/forms/PQTvWKjHgg96DjD8/fill> 欢迎 openEuler 社区的开发者提交以上四个方面的材料。 [image: 截屏2021-05-13 下午8.12.43.png] *大会亮点* 这是一场大型的面向 openEuler 社区开发者的技术交流会议。在现场共同探讨*热点技术方向*和*如何参与社区贡献*，通过大会*公开接收用户需求*，*完成版本规划*。 *技术委员会*、*品牌委员会*、*用户委员会*、*20个 SIG 组*会同期举办*开放工作会议**（仅限签署社区 CLA 的开发者参加）*。技术委员会主席给大家解读 *openEuler 21.03*，发布下个社区版的*技术定位*，分享社区创新思路。来自互联网、金融、运营商的客户分享成功的应用实践经验。开源圆桌中，客户、伙伴、厂商、研究机构一同探究开源软件供应链的意义和风险，开源模式如何推动独立演进操作系统技术的发展。更有开发者心心念念的 *Party* 心动不如行动，赶紧来报名参会 <https://e-campaign.huawei.com/m/mIzQFr>吧！！！

1 0

[PATCH v4 lkp-tests 7/7] distro/depends: add install depend for benchmarktool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- distro/depends/mysql-8-0-20-dev | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/mysql-8-0-20-dev b/distro/depends/mysql-8-0-20-dev index 23c6ff661..05583bef9 100644 --- a/distro/depends/mysql-8-0-20-dev +++ b/distro/depends/mysql-8-0-20-dev @@ -14,3 +14,4 @@ libmpc libmpc-devel mpfr mpfr-devel +java -- 2.23.0

1 0

[PATCH v4 lkp-tests 6/7] pkg/mysql-8-0-20: add build install benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- pkg/mysql-8-0-20/PKGBUILD | 51 ++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/pkg/mysql-8-0-20/PKGBUILD b/pkg/mysql-8-0-20/PKGBUILD index 66df6b33f..26247c6f0 100644 --- a/pkg/mysql-8-0-20/PKGBUILD +++ b/pkg/mysql-8-0-20/PKGBUILD @@ -7,13 +7,14 @@ url="http://www.mysql.com" license=('GPLv2') source=("git://$GIT_SERVER/upstream/m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch.git" "git://$GIT_SERVER/upstream/g/gcc/gcc.git" + "https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/database/p…" "my.cnf") -md5sums=('SKIP' 'SKIP' '99c4965aa870c475000993f52a0dee66') +md5sums=('SKIP' 'SKIP' 'f33c03fd48ff5993dca6d00b53a46da4' '99c4965aa870c475000993f52a0dee66') build() { - build_gcc build_mysql + build_benchmarksql } build_gcc() @@ -47,9 +48,8 @@ build_gcc() --with-stage1-ldflags='-Wl,-z,relro,-z,now' \ --with-boot-ldflags='-Wl,-z,relro,-z,now' \ --with-multilib-list=lp64 - make -j96 - make install & - wait + make -j96 > /dev/null + make install > /dev/null } } @@ -58,6 +58,11 @@ build_mysql() unset CPPFLAGS unset CFLAGS unset CXXFLAGS + + cd $srcdir/my-sql_with_boostkit_patch && { + grep 3f3d2692fd05a18011d93dcadeef661f386595ac compass-ci.cfg || build_gcc + } + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver CMAKE=cmake @@ -65,8 +70,36 @@ build_mysql() mkdir -p build cd build && { - $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ - make -j96 + $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ > /dev/null + make -j96 > /dev/null + } +} + +build_benchmarksql() +{ + cd $srcdir && { + cp -r benchmarksql5.0-for-mysql "$HOME/" + chmod -R 777 "$HOME/benchmarksql5.0-for-mysql" + } + + cd "$HOME/benchmarksql5.0-for-mysql/run" && { + chmod 777 *.sh + sed -i \ + -e '/^conn=/ s|mysql://.*:3306|mysql://localhost:3306|' \ + -e '/^user=/cuser=root' \ + -e '/^password=/cpassword=mysql_password' \ + -e '/^warehouses=/cwarehouses=1000' \ + -e '/^loadWorkers=/cloadWorkers=100' \ + -e '/^terminals=/cterminals=200' \ + -e '/^runTxnsPerTerminal=/crunTxnsPerTerminal=0' \ + -e '/^runMins=/crunMins=30' \ + -e '/^limitTxnsPerMin=/climitTxnsPerMin=1000' \ + -e '/^terminalWarehouseFixed=/cterminalWarehouseFixed=true' \ + -e '/^newOrderWeight=/cnewOrderWeight=45' \ + -e '/^paymentWeight=/cpaymentWeight=43' \ + -e '/^orderStatusWeight=/corderStatusWeight=4' \ + -e '/^deliveryWeight=/cdeliveryWeight=4' \ + props.conf } } @@ -75,6 +108,6 @@ package() cp my.cnf /etc/ cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver/build && { - make install - } + make install > /dev/null + } } -- 2.23.0

1 0

[PATCH v4 lkp-tests 5/7] tests: add init database for benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 66 +++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 30 deletions(-) diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 index 990a053d7..4a49e0233 100755 --- a/tests/mysql-8-0-20 +++ b/tests/mysql-8-0-20 @@ -4,69 +4,75 @@ # mysql_host # mysql_port # mysql_db +# benchmark_db # mysql_password + . $LKP_SRC/lib/common.sh : "${mysql_user:=root}" : "${mysql_host:=127.0.0.1}" : "${mysql_port:=3306}" : "${mysql_db:=sysbench_test}" +: "${benchmark_db:=tpcc}" : "${mysql_password:=mysql_password}" create_basedir() { - mkdir -p /data - mkfs.xfs -f /dev/"$data_dev" - mount /dev/"$data_dev" /data + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data - mkdir -p /data/mysql/{data,share,tmp,run,log} + mkdir -p /data/mysql/{data,share,tmp,run,log} - useradd mysql - touch /data/mysql/log/mysql.log - chown -R mysql:mysql /data - chown -R mysql:mysql /data/mysql/log/mysql.log + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log } configure_mysql() { - if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then - echo "mysql is not installed!" && exit 1 - fi + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi - "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" - sleep 10 + "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" + sleep 10 - "$mysql_server" start - sleep 10 + "$mysql_server" start + sleep 10 - "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" - "$mysql_server" restart - sleep 10 + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + sed -i '/^skip-name-resolve/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 } create_database() { - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" - "$mysql_server" stop - sleep 10 + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $benchmark_db;" + "$mysql_server" stop + sleep 10 } start_mysql() { - "$mysql_server" start & + "$mysql_server" start & wait } main() { - local mysql="/usr/local/mysql/bin/mysql" - local mysqld="/usr/local/mysql/bin/mysqld" - local mysql_server="/usr/local/mysql/support-files/mysql.server" + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" - create_basedir - configure_mysql - create_database - start_mysql + create_basedir + configure_mysql + create_database + start_mysql } main -- 2.23.0

1 0

[PATCH v4 lkp-tests 4/7] tests: add script for benchmarksql test tool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/benchmark-sql | 15 +++++++++++++++ 1 file changed, 15 insertions(+) create mode 100755 tests/benchmark-sql diff --git a/tests/benchmark-sql b/tests/benchmark-sql new file mode 100755 index 000000000..6acac2847 --- /dev/null +++ b/tests/benchmark-sql @@ -0,0 +1,15 @@ +#!/bin/sh +# benchmark_pkg_name + +: "${benchmark_pkg_name:=benchmarksql5.0-for-mysql}" + +run_benchmark_mysql() +{ + cd "$HOME/$benchmark_pkg_name/run/" && { + ./runDatabaseBuild.sh props.conf + ./runBenchmark.sh props.conf + ./runDatabaseDestroy.sh props.conf + } +} + +run_benchmark_mysql -- 2.23.0

1 0

[PATCH v4 lkp-tests 3/7] stats: add extract scritp
by Luan Shengde 30 Apr '21

30 Apr '21

add stats script for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- stats/benchmark-sql | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 stats/benchmark-sql diff --git a/stats/benchmark-sql b/stats/benchmark-sql new file mode 100755 index 000000000..d4794b4f4 --- /dev/null +++ b/stats/benchmark-sql @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby + +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmC (NewOrders) = 450.31 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmTOTAL = 999.71 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session Start = 2021-04-29 10:58:15 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session End = 2021-04-29 11:28:16 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Transaction Count = 30003 + +while (line = STDIN.gets) + case line.chomp! + when /Measured tpmC $NewOrders$ = (\d*\.\d*)/ + puts "Measured_tpmC$NewOrders$: #{$1}" + when /Measured tpmTOTAL = (\d*\.\d*)/ + puts "Measured_tpmTOTAL: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v4 lkp-tests 2/7] jobs: add job yaml
by Luan Shengde 30 Apr '21

30 Apr '21

add job yaml for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/benchmark-sql.yaml | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 jobs/benchmark-sql.yaml diff --git a/jobs/benchmark-sql.yaml b/jobs/benchmark-sql.yaml new file mode 100644 index 000000000..4563b666c --- /dev/null +++ b/jobs/benchmark-sql.yaml @@ -0,0 +1,16 @@ +suite: benchmark-sql + +mysql_user: +mysql_host: localhost +mysql_port: +mysql_db: +benchmark_db: tpcc +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +benchmark-sql: -- 2.23.0

1 0

[PATCH v4 lkp-tests 1/7] jobs: add job yaml
by Luan Shengde 30 Apr '21

30 Apr '21

add job yaml for sysbench-mysql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/sysbench-mysql.yaml | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) create mode 100644 jobs/sysbench-mysql.yaml diff --git a/jobs/sysbench-mysql.yaml b/jobs/sysbench-mysql.yaml new file mode 100644 index 000000000..af017df4b --- /dev/null +++ b/jobs/sysbench-mysql.yaml @@ -0,0 +1,28 @@ +suite: sysbench-mysql + +mysql_user: +mysql_host: localhost +mysql_port: +mysql_db: +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +sysbench-mysql: + oltp_test_mode: complex + max_requests: 0 + mysql_table_engine: innodb + oltp_tables_count: 1000 + oltp_table_size: 100000 + nr_threads: 256 + runtime: 60 + rand_type: special + rand_spec_pct: 100 + oltp_read_only: false + report_interval: 1 + events: 20000 + mysql_socket: /data/mysql/run/mysql.sock -- 2.23.0

1 0

[PATCH v2 lkp-tests] resize virtual rootfs from 50% to 90% memory, for docker no sapce error
by Zhang Dewan 30 Apr '21

30 Apr '21

Signed-off-by: Zhang Dewan <513619464(a)qq.com> --- lib/bootstrap.sh | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/lib/bootstrap.sh b/lib/bootstrap.sh index 417c8662a..82a1c17d7 100755 --- a/lib/bootstrap.sh +++ b/lib/bootstrap.sh @@ -57,6 +57,35 @@ mount_tmpfs() mount -t tmpfs -o mode=1777 tmp /tmp } +resize_virtual_rootfs() +{ + # tmpfs and rootfs default use 50% memory: + # + # root@taishan200-2280-2s48p-256g--a13 ~# mount | head -n 1 + # rootfs on / type rootfs (rw,size=133395136k,nr_inodes=2084299) + # + # root@taishan200-2280-2s48p-256g--a13 ~# df -h / + # Filesystem Size Used Avail Use% Mounted on + # rootfs 128G 114G 14G 90% / + # + # this maybe cause the docker no space error. + + if mount | head -n 1 | grep "type rootfs" > /dev/null + then + echo "Set the rootfs to 90% memory" + mount -o remount,rw,size=90% rootfs / + fi + # Test Result: + # root@taishan200-2280-2s64p-256g--a125 ~# free -g + # total used free shared buff/cache available + # Mem: 255Gi 8.5Gi 243Gi 2.7Gi 2.9Gi 224Gi + # Swap: 0B 0B 0B + # root@taishan200-2280-2s64p-256g--a125 ~# df -h / + # Filesystem Size Used Avail Use% Mounted on + # rootfs 230G 2.7G 228G 2% / + +} + get_net_devices() { local net_devices @@ -737,6 +766,7 @@ boot_init() mount_kernel_fs mount_tmpfs + resize_virtual_rootfs redirect_stdout_stderr echo 'Kernel tests: Boot OK!' -- 2.23.0

1 0

[PATCH compass-ci] sparrow/README.md: specify the correct filename in source cmd
by Xu Xijian 30 Apr '21

30 Apr '21

Signed-off-by: Xu Xijian <hdxuxijian(a)163.com> --- sparrow/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sparrow/README.md b/sparrow/README.md index 532b7f9..a519497 100644 --- a/sparrow/README.md +++ b/sparrow/README.md @@ -61,7 +61,7 @@ 本文以/c/lkp-tests/jobs/目录下已有的通用测试用例host-info.yaml为例 - 使环境变量生效 ```bash - source /etc/profile.d/compass-ci + source /etc/profile.d/compass.sh ``` - 使用[submit命令](https://gitee.com/wu_fengguang/compass-ci/blob/master/doc/manu… -- 2.23.0

1 0

[PATCH v3 lkp-tests 7/7] distro/depends: add depend for benchmarktool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- distro/depends/mysql-8-0-20-dev | 1 + 1 file changed, 1 insertion(+) diff --git a/distro/depends/mysql-8-0-20-dev b/distro/depends/mysql-8-0-20-dev index 23c6ff661..05583bef9 100644 --- a/distro/depends/mysql-8-0-20-dev +++ b/distro/depends/mysql-8-0-20-dev @@ -14,3 +14,4 @@ libmpc libmpc-devel mpfr mpfr-devel +java -- 2.23.0

1 0

[PATCH v3 lkp-tests 6/7] pkg/mysql-8-0-20: add build install benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- pkg/mysql-8-0-20/PKGBUILD | 51 ++++++++++++++++++++++++++++++++------- 1 file changed, 42 insertions(+), 9 deletions(-) diff --git a/pkg/mysql-8-0-20/PKGBUILD b/pkg/mysql-8-0-20/PKGBUILD index 66df6b33f..26247c6f0 100644 --- a/pkg/mysql-8-0-20/PKGBUILD +++ b/pkg/mysql-8-0-20/PKGBUILD @@ -7,13 +7,14 @@ url="http://www.mysql.com" license=('GPLv2') source=("git://$GIT_SERVER/upstream/m/my-sql_with_boostkit_patch/my-sql_with_boostkit_patch.git" "git://$GIT_SERVER/upstream/g/gcc/gcc.git" + "https://mirrors.huaweicloud.com/kunpeng/archive/kunpeng_solution/database/p…" "my.cnf") -md5sums=('SKIP' 'SKIP' '99c4965aa870c475000993f52a0dee66') +md5sums=('SKIP' 'SKIP' 'f33c03fd48ff5993dca6d00b53a46da4' '99c4965aa870c475000993f52a0dee66') build() { - build_gcc build_mysql + build_benchmarksql } build_gcc() @@ -47,9 +48,8 @@ build_gcc() --with-stage1-ldflags='-Wl,-z,relro,-z,now' \ --with-boot-ldflags='-Wl,-z,relro,-z,now' \ --with-multilib-list=lp64 - make -j96 - make install & - wait + make -j96 > /dev/null + make install > /dev/null } } @@ -58,6 +58,11 @@ build_mysql() unset CPPFLAGS unset CFLAGS unset CXXFLAGS + + cd $srcdir/my-sql_with_boostkit_patch && { + grep 3f3d2692fd05a18011d93dcadeef661f386595ac compass-ci.cfg || build_gcc + } + cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver CMAKE=cmake @@ -65,8 +70,36 @@ build_mysql() mkdir -p build cd build && { - $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ - make -j96 + $CMAKE .. -DBUILD_CONFIG=mysql_release -DCMAKE_INSTALL_PREFIX=/usr/local/mysql -DMYSQL_DATADIR=/data/mysql/data -DSYSCONFDIR=/etc -DWITH_BOOST=../boost/boost_1_70_0/ > /dev/null + make -j96 > /dev/null + } +} + +build_benchmarksql() +{ + cd $srcdir && { + cp -r benchmarksql5.0-for-mysql "$HOME/" + chmod -R 777 "$HOME/benchmarksql5.0-for-mysql" + } + + cd "$HOME/benchmarksql5.0-for-mysql/run" && { + chmod 777 *.sh + sed -i \ + -e '/^conn=/ s|mysql://.*:3306|mysql://localhost:3306|' \ + -e '/^user=/cuser=root' \ + -e '/^password=/cpassword=mysql_password' \ + -e '/^warehouses=/cwarehouses=1000' \ + -e '/^loadWorkers=/cloadWorkers=100' \ + -e '/^terminals=/cterminals=200' \ + -e '/^runTxnsPerTerminal=/crunTxnsPerTerminal=0' \ + -e '/^runMins=/crunMins=30' \ + -e '/^limitTxnsPerMin=/climitTxnsPerMin=1000' \ + -e '/^terminalWarehouseFixed=/cterminalWarehouseFixed=true' \ + -e '/^newOrderWeight=/cnewOrderWeight=45' \ + -e '/^paymentWeight=/cpaymentWeight=43' \ + -e '/^orderStatusWeight=/corderStatusWeight=4' \ + -e '/^deliveryWeight=/cdeliveryWeight=4' \ + props.conf } } @@ -75,6 +108,6 @@ package() cp my.cnf /etc/ cd $srcdir/my-sql_with_boostkit_patch/$pkgname-$pkgver/build && { - make install - } + make install > /dev/null + } } -- 2.23.0

1 0

[PATCH v3 lkp-tests 5/7] tests: add init database for benchmarksql
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/mysql-8-0-20 | 66 +++++++++++++++++++++++++--------------------- 1 file changed, 36 insertions(+), 30 deletions(-) diff --git a/tests/mysql-8-0-20 b/tests/mysql-8-0-20 index 990a053d7..4a49e0233 100755 --- a/tests/mysql-8-0-20 +++ b/tests/mysql-8-0-20 @@ -4,69 +4,75 @@ # mysql_host # mysql_port # mysql_db +# benchmark_db # mysql_password + . $LKP_SRC/lib/common.sh : "${mysql_user:=root}" : "${mysql_host:=127.0.0.1}" : "${mysql_port:=3306}" : "${mysql_db:=sysbench_test}" +: "${benchmark_db:=tpcc}" : "${mysql_password:=mysql_password}" create_basedir() { - mkdir -p /data - mkfs.xfs -f /dev/"$data_dev" - mount /dev/"$data_dev" /data + mkdir -p /data + mkfs.xfs -f /dev/"$data_dev" + mount /dev/"$data_dev" /data - mkdir -p /data/mysql/{data,share,tmp,run,log} + mkdir -p /data/mysql/{data,share,tmp,run,log} - useradd mysql - touch /data/mysql/log/mysql.log - chown -R mysql:mysql /data - chown -R mysql:mysql /data/mysql/log/mysql.log + useradd mysql + touch /data/mysql/log/mysql.log + chown -R mysql:mysql /data + chown -R mysql:mysql /data/mysql/log/mysql.log } configure_mysql() { - if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then - echo "mysql is not installed!" && exit 1 - fi + if [ ! -f $mysqld ] || [ ! -f $mysql ] || [ ! -f $mysql_server ]; then + echo "mysql is not installed!" && exit 1 + fi - "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" - sleep 10 + "$mysqld" --user=mysql --datadir=/data/mysql/data --basedir=/data/mysql --initialize --port="$mysql_port" + sleep 10 - "$mysql_server" start - sleep 10 + "$mysql_server" start + sleep 10 - "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" - "$mysql_server" restart - sleep 10 + "$mysql" -u "$mysql_user" -e "flush privileges; use mysql; alter user 'root'@'localhost' identified by '$mysql_password'; flush privileges;" + sed -i '/^skip-grant-tables/d' /etc/my.cnf + sed -i '/^skip-name-resolve/d' /etc/my.cnf + "$mysql_server" restart + sleep 10 } create_database() { - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" - "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" - "$mysql_server" stop - sleep 10 + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "use mysql; update user set host ='%' where user='root'; flush privileges;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $mysql_db;" + "$mysql" -u "$mysql_user" -p"$mysql_password" -e "create database $benchmark_db;" + "$mysql_server" stop + sleep 10 } start_mysql() { - "$mysql_server" start & + "$mysql_server" start & wait } main() { - local mysql="/usr/local/mysql/bin/mysql" - local mysqld="/usr/local/mysql/bin/mysqld" - local mysql_server="/usr/local/mysql/support-files/mysql.server" + local mysql="/usr/local/mysql/bin/mysql" + local mysqld="/usr/local/mysql/bin/mysqld" + local mysql_server="/usr/local/mysql/support-files/mysql.server" - create_basedir - configure_mysql - create_database - start_mysql + create_basedir + configure_mysql + create_database + start_mysql } main -- 2.23.0

1 0

[PATCH v3 lkp-tests 4/7] tests: add script for benchmarksql test tool
by Luan Shengde 30 Apr '21

30 Apr '21

Signed-off-by: Luan Shengde <shdluan(a)163.com> --- tests/benchmark-sql | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) create mode 100755 tests/benchmark-sql diff --git a/tests/benchmark-sql b/tests/benchmark-sql new file mode 100755 index 000000000..2ec6a7317 --- /dev/null +++ b/tests/benchmark-sql @@ -0,0 +1,17 @@ +#!/bin/sh +# benchmark_pkg_name + +: "${benchmark_pkg_name:=benchmarksql5.0-for-mysql}" + +run_benchmark_mysql() +{ + scripts=(runDatabaseBuild.sh runBenchmark.sh runDatabaseDestroy.sh) + + cd "$HOME/$benchmark_pkg_name/run/" && { + ./runDatabaseBuild.sh props.conf + ./runBenchmark.sh props.conf + ./runDatabaseDestroy.sh props.conf + } +} + +run_benchmark_mysql -- 2.23.0

1 0

[PATCH v3 lkp-tests 3/7] stats: add extract scritp
by Luan Shengde 30 Apr '21

30 Apr '21

add stats script for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- stats/benchmark-sql | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100755 stats/benchmark-sql diff --git a/stats/benchmark-sql b/stats/benchmark-sql new file mode 100755 index 000000000..d4794b4f4 --- /dev/null +++ b/stats/benchmark-sql @@ -0,0 +1,16 @@ +#!/usr/bin/env ruby + +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmC (NewOrders) = 450.31 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Measured tpmTOTAL = 999.71 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session Start = 2021-04-29 10:58:15 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Session End = 2021-04-29 11:28:16 +# 11:28:16,052 [Thread-2] INFO jTPCC : Term-00, Transaction Count = 30003 + +while (line = STDIN.gets) + case line.chomp! + when /Measured tpmC $NewOrders$ = (\d*\.\d*)/ + puts "Measured_tpmC$NewOrders$: #{$1}" + when /Measured tpmTOTAL = (\d*\.\d*)/ + puts "Measured_tpmTOTAL: #{$1}" + end +end -- 2.23.0

1 0

[PATCH v3 lkp-tests 2/7] jobs: add job yaml
by Luan Shengde 30 Apr '21

30 Apr '21

add job yaml for benchmarksql Signed-off-by: Luan Shengde <shdluan(a)163.com> --- jobs/benchmark-sql.yaml | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) create mode 100644 jobs/benchmark-sql.yaml diff --git a/jobs/benchmark-sql.yaml b/jobs/benchmark-sql.yaml new file mode 100644 index 000000000..152253c4e --- /dev/null +++ b/jobs/benchmark-sql.yaml @@ -0,0 +1,20 @@ +suite: benchmark-sql + +os: openeuler +os_version: 20.03 +os_mount: initramfs + +mysql_user: +mysql_host: localhost +mysql_port: +mysql_db: +benchmark_db: tpcc +mysql_password: +data_dev: sdb + +cci-makepkg: + benchmark: mysql-8-0-20 + +mysql-8-0-20: + +benchmark-sql: -- 2.23.0

1 0