February 2022 - Compass-ci - mailweb.openeuler.org

Re: [TODO] provide LKP dependencies in container with busybox+curl
by Luan Shengde 09 Feb '22

09 Feb '22

- use debian OS as host OS - make sure host installed busybox-static package and downloaded static-curl - mount busybox/curl into docker - create busybox symlinks in docker mkdir /opt/busybox/bin/ cd /opt/busybox/bin/ for c in wget cpio do ln -s /path/to/busybox $c done PATH=$PATH:/opt/busybox/bin/ by fengguang thanks - download .cgz depends inside docker in LKP framework - extract .cgz On Mon, Feb 07, 2022 at 02:26:54PM +0800, Wu Fengguang wrote: >Currently /c/compass-ci/providers/docker/run.sh has extra dependencies >that should be simplified. > >TODO1: install busybox in the host (I see no script to make sure it's >installed in the host). If the host OS does not has the busybox >package, it's fine to download pre-built busybox-static binary > >TODO2: make sure busybox symlinks are created in a dir like >/opt/busybox/bin/ inside the container, and _append_ it to PATH. > >TODO3: install curl-static in the host and mount into container >/opt/curl/bin/ > > https://github.com/moparisthebest/static-curl > > Grab the latest release from one of these links, by CPU architecture: > curl-amd64 > curl-i386 > curl-aarch64 > curl-armv7 > curl-ppc64le > >busybox+curl should provide all the *required* dependencies for >running LKP framework. > >Thanks, >Fengguang > >On Tue, Jan 04, 2022 at 10:07:08AM +0800, Wu Fengguang wrote: >>Let's cut down dependencies in /c/lkp-tests/distro/depends/lkp >> >>For the depends added by >> >> commit 54e515a10156080bfabc8d880b3f704b7c29a702 >> Author: Wei Jihui <weijihuiall(a)163.com> >> AuthorDate: Mon Nov 8 09:59:15 2021 +0800 >> Commit: Wei Jihui <weijihuiall(a)163.com> >> CommitDate: Thu Nov 11 14:25:03 2021 +0800 >> >> distro/depends/lkp: move cci-depends'depends to lkp's depends >> >>we can use either >>- busybox ftpput >>- rsync >>- lftp >>for uploading files. Those packages are rarely depended by normal >>packages. In comparison, curl is depended by hundreds of other >>packages in Debian. >> >>If use busybox, lkp can remove lots of dependencies. >> >>However for remote testbox, its 'busybox ftpput' command may not work well >>- need PASSIVE mode in order to go through firewall >>- not secure over Internet >>- some firewall may bann insecure service/ports >>- not good to block testbox in slow/unreliable Internet connection >> >>The same for rsync upload, if ssh is not used together. >> >>But shall we upload all remote testbox results over Internet? >>I guess not -- it won't scale. >> >>Here is a flexible upload model >> >>testbox >>-> on job finish, tar/cpio all results into a single file >> (to avoid complex mirror function for ftpput) >>-> upload via any of ftp/rsync/https to a dedicated and temp server storage space >> (eg. /srv/result/upload/$jobid/ with md5sum verification) >> (proftpd: configure as upload only dir, disable read and list-dir) >>-> tell scheduler that the job is complete >>-> run a service to unpack a job's results into final result_root >> >>remote testbox >>-> results uploaded to a local server storage >>-> local storage server runs a service to send results to central lab storage >> when asked by end user >> >>Thanks, >>Fengguang

1 0

extract buildinfo
by Lu Weitao 09 Feb '22

09 Feb '22

# extract buildinfo and save it to ES todo from reproducible-build flow ``` after build rpm, and upload buildinfo.yaml, rpmbuild to server | v $result_root/rpmbuild.yaml ---> rpmbuild.json ---> merge to stats.json $result_root/rpmbuild_info.yaml ---> rpmbuild_info.json ---> merge to stats.json | v save stats.json to ES db # 为compare， diifoscope 提供数据源 eg: { "id": xxx, "os": "openeuler", "arch": "aarch64" "stats": { "rpmbuild.hash_value": xxx, ``` ## TODO1: improve compass-ci/sbin/result2stats, extract-stats service call it support extract buildinfo usage: result2stats $result_root ``` ... $result_root/rpmbuild ---> rpmbuild.json ---> merge to stats.json $result_root/buildinfo.yaml ---> buildinfo.json ---> merge to stats.json ... ``` ### input: result_root/buildinfo(buildinfo.yaml) - e.g. samba_4.9.5+dfsg-5+deb10u2_arm64.buildinfo ``` Format: 1.0 Source: samba Binary: samba samba-libs samba-common samba-common-bin smbclient samba-testsuite registry-tools samba-dev python-samba samba-dsdb-modules samba-vfs-modules libsmbclient libsmbclient-dev winbind libpam-winbind libnss-winbind libwbclient0 libwbclient-dev ctdb Architecture: all arm64 source Version: 2:4.9.5+dfsg-5+deb10u2 Checksums-Md5: 79ffac5466e2b36633e0f429644407d0 3211 samba_4.9.5+dfsg-5+deb10u2.dsc d523774b75cc4d31dce4696d41d68b8f 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Checksums-Sha1: b8c10499c1b6b7c5111c7f2d8d617ed8a2ff938e 3211 samba_4.9.5+dfsg-5+deb10u2.dsc 1569bfdb4399f92852863f67153cce046a0cbb38 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Checksums-Sha256: c9094fcc63930f665707c596134ea64e44d433d3882b3de544c2a83f574aafb9 3211 samba_4.9.5+dfsg-5+deb10u2.dsc 3304c2fa4f7204682747bed70f7ce61b134b21f063f9ebab0aeb2e3af645b744 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb ... Build-Origin: Debian Build-Architecture: arm64 Build-Date: Sun, 30 Jan 2022 09:27:50 +0000 Installed-Build-Depends: adduser (= 3.118), zlib1g-dev (= 1:1.2.11.dfsg-1) ... Environment: DEB_BUILD_OPTIONS="parallel=128" SOURCE_DATE_EPOCH="1638005690" ``` ### output result_root/buildinfo.json ``` { "Hash": "SHA512", "Format": 1.0, "Build-Architecture": "i386", "Source": "fweb (1.62-12)", "Binary": "fweb fweb-doc", "Architecture": "all i386", "Version": "1.62-12+b2", "Binary-Only-Changes": "fweb (1.62-12+b2) sid; urgency=low, binary-only=yes . * Binary-only non-maintainer upload for amd64; no source changes. * Rebuild for multiarch sync . -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms(a)buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000", ... } ``` ## TODO2: save buildinfo to job(ES) extract-service load $result_root/buildinfo.json - option1: merge job['buildinfo'](job['record']['buildinfo']) meeting record: job.record.build.xxx: v => possible solution option1 通用性好（job['record']['buildinfo'][xxx]）结构较复杂 job.info.build.xxx: v 效果同上 value type: - output binary hash - info (xxx.rpm) job['buildinfo'] 结构简单，但只适用与构建任务，目前未发现其它类型的job有这种需求，如需使用buildinfo进行搜索，那这种方式更简单 **job like:** ``` { "id": xxx, "os": "openeuler", ... "buildinfo": { "Hash": "SHA512", "Format": 1.0, ... } } ``` **advantage:** 1. reuse job['buildinfo'] to submit again # we can also merge it to job.yaml 2. 3rd tool such as: `diffoscope` get job['buildinfo'] **disadvantage:** 1. compare need get metric from buildinfo in addition - option2: merge buildinfo to job['stats'], job like: ``` { "id": xxx, "os": "openeuler", "stats": { "buildinfo.Hash": "SHA512", "buildinfo.Format": 1.0, ... } } ``` advantage / disadvantage: opposite of option1 job.stats.rpmbuild.xxx job['stats']['rpmbuild.xxx'] compare 不好区分buildinfo 和 rpmbuild 需要变动lkp-tests/stats/rpmbuild，且buildinfo > result_root/rpmbuild job.stats.buildinfo.xxx => possible solution option2 job['stats']['buildinfo.xxx'] extract-stats 已经支持，无需改动代码仅仅将buildinfo信息输出未 result_root/buildinfo.yaml compare 性能结果时忽略 buildinfo （可以新加perf-metric-ignore规则） job.stats.message.xxx / job.stats.xxx.message 目前.message, .log 已经是perf-metric-ignore规则，可以考虑将buildinfo也加进去 job.stats.info.build.buildinfo.xhhhhhxx => possible solution 通用性更好，相当于对metric有个全面的分类结构较多 job.buildinfo.xxx（in option1） - option3: option1 + option2 job like: ``` { "id": xxx, "os": "openeuler", ... "buildinfo": { "Hash": "SHA512", "Format": 1.0, ... }, "stats": { "buildinfo.Hash": "SHA512", "buildinfo.Format": 1.0, ... } } ``` **advantage: opton1 + option2 ** **disadvantage:** 1. job bigger, --> save DB 2. extract-service does one more thing

1 0

Re: reproducible-build TODO list
by Lu Weitao 09 Feb '22

09 Feb '22

On Tue, Feb 08, 2022 at 08:20:37PM +0800, Wei Jihui wrote: >On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >>## TODO list >>- TODO1.1: script support set env for build env --- compass-ci jihui > >what should i do for this? > user submit rpmbuild.yaml will with some build params like: fs: kernel: reproducible_build_env: hostname: datetime: rpmbuild: new_param: we should make sure these params take effect in testbox --------- Thanks, Lu Weitao - >Thanks, >Jihui > > >> >>- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci >> - based on debian buildinfo >> - TODO1.3: 输出用户指定的息into buildinfo? >> new lib/buildinfo.sh --- IT >> func1() >> func2() >> func3() >> save to $result_root/buildinfo.yaml >> >> modify rpmbuild script to call the above funcs. --- compass-ci wangyong >> >>- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao >> >>- TODO2: 比较构建结果是否相同 --- compass-ci weitao >> compass ci >> >> reuse data API es-find >> compare in client side >> >> 格式和进一步优化: IT >> client: ask use python? >> >>- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT >> >>- TODO4 toolset --- IT/compass-ci >> option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 >> option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT >> option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong >> >> >> >>--------- >>Thanks, >>Lu Weitao >>

1 0

Re: [PATCH lkp-tests 3/3] lkp-tests/*: add new flow for install lkp depends
by Wang Yong 09 Feb '22

09 Feb '22

On Tue, Feb 08, 2022 at 11:28:38AM +0800, Wei Jihui wrote: >On Sun, Jan 30, 2022 at 08:58:23AM +0800, Wang Yong wrote: >>[why] >>install lkp depends before test suite to satisfy lkp-tests >> >>Signed-off-by: Wang Yong <wangyong187(a)huawei.com> >>--- >> lib/job2sh.rb | 2 ++ >> setup/install_depends | 49 +++++++++++++++++++++++++++++++++++-------- >> 2 files changed, 42 insertions(+), 9 deletions(-) >> >>diff --git a/lib/job2sh.rb b/lib/job2sh.rb >>index 4e8bf97fb..32caf2ade 100755 >>--- a/lib/job2sh.rb >>+++ b/lib/job2sh.rb >>@@ -316,6 +316,7 @@ class Job2sh < Job >> @monitors = available_programs(:monitors) >> @setups = available_programs(:setup) >> @programs = available_programs(:workload_elements) >>+ @lkp = {"install_depends"=>{}} > >how about use '@install_depends'? > ok Thanks, Yong

1 0

Re: reproducible-build TODO list
by weijihui 08 Feb '22

08 Feb '22

On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >## TODO list >- TODO1.1: script support set env for build env --- compass-ci jihui what should i do for this? Thanks, Jihui > >- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci > - based on debian buildinfo > - TODO1.3: 输出用户指定的息into buildinfo? > new lib/buildinfo.sh --- IT > func1() > func2() > func3() > save to $result_root/buildinfo.yaml > > modify rpmbuild script to call the above funcs. --- compass-ci wangyong > >- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao > >- TODO2: 比较构建结果是否相同 --- compass-ci weitao > compass ci > > reuse data API es-find > compare in client side > > 格式和进一步优化: IT > client: ask use python? > >- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT > >- TODO4 toolset --- IT/compass-ci > option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 > option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT > option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong > > > >--------- >Thanks, >Lu Weitao >

1 0

Re: [PATCH compass-ci] container/data-api/data-api: fix incorrect comments
by Lu Weitao 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 09:39:14AM +0800, Wei Jihui wrote: >On Mon, Feb 07, 2022 at 11:22:13AM +0800, Lu Weitao wrote: >>Signed-off-by: Lu Weitao <luweitaobe(a)163.com> >>--- >> container/data-api/data-api | 4 ++-- >> 1 file changed, 2 insertions(+), 2 deletions(-) >> >>diff --git a/container/data-api/data-api b/container/data-api/data-api >>index 2bd6e527..c1c8af8f 100755 >>--- a/container/data-api/data-api >>+++ b/container/data-api/data-api >>@@ -17,7 +17,7 @@ set :environment, :production >> >> # POST >> # eg1: >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es_find -H 'Content-Type: application/json' -d '{ > > >we will define new api name for data_api. > ok --------- Thanks, Lu Weitao >Thanks, >Jihui > > >> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >> # "index": "jobs", >> # "my_token": "16132550-...", >>@@ -25,7 +25,7 @@ set :environment, :production >> # "query_type": "eql" # can be omitted in this scenario >> # }' >> # eg2: >>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>+# curl -X POST localhost:10005/data_api/es_find -H 'Content-Type: application/json' -d '{ >> # "query": "SELECT * FROM jobs WHERE ...", >> # "index": "jobs", >> # "my_token": "16132550-...", >>-- >>2.23.0 >>

1 0

Re: [PATCH v2 compass-ci] data-api: add es search api
by Wei Jihui 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 10:28:21AM +0800, Wu Fengguang wrote: >>+ def self.search(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] || {'query' => {}} >>+ >>+ return "index is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) > >这里的return会在上一级被作为HTTP 200 >但看起来是一个错误情况。是的，抛出错误更好一些。 Thanks, Jihui > >>+ def self.search_by_sql(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] >>+ >>+ return "query table is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) > >同上。 > >Thanks, >Fengguang >

1 0

Re: [PATCH v2 compass-ci] data-api: add es search api
by Wei Jihui 08 Feb '22

08 Feb '22

On Tue, Feb 08, 2022 at 10:24:24AM +0800, Wu Fengguang wrote: >>+module EsDataApi >>+ ES_ACCOUNTS = ESClient.new(index: 'accounts') >>+ OPEN_INDEX = Set.new(['jobs']) >>+ REQUIRED_TOKEN_INDEX = Set.new(['jobs']) >>+ ES_QUERY_KEYWORD = Set.new(['term', 'match']) >>+ >>+ > >2 empty lines can be merged. ok. > >>+ def self.search(index, params) >>+ request_body = JSON.parse(params) >>+ query = request_body['query'] || {'query' => {}} >>+ >>+ return "index is empty!" if index.nil? >>+ return "#{index} is not opened" unless OPEN_INDEX.include?(index) >>+ >>+ if REQUIRED_TOKEN_INDEX.include?(index) >>+ cci_authorization = request_body['cci_authorization'] || {} >>+ my_account = cci_authorization['my_account'] >>+ my_token = cci_authorization['my_token'] >>+ raise "user authentication failed" unless verify_user(my_account, my_token) > >加上my_account，把错误信息变得友好些。 ok. user authentication failed，please check my_account and my_token. > >>+ > >函数体有点大。功能型代码块，适合独立拆分出去。 ok. >>+ def self.verify_user(my_account, my_token) >>+ raise "missed my_account" unless my_account >>+ raise "missed my_token" unless my_token >>+ >>+ query = {} >>+ query['my_account'] = my_account if my_account >>+ query['my_token'] = my_token if my_token > >这里的两个if前面判断过了，可省略 > ok Thanks, Jihui >>+ result = ES_ACCOUNTS.multi_field_query(query)['hits']['hits'] >>+ return nil if result.size == 0 >>+ >>+ result[0]['_source']['my_token'] == my_token >>+ end >>+end >>-- >>2.23.0 >>

1 0

Re: [PATCH compass-ci] src/lib/web_backend.rb: set page_num and page_size to a default value.
by Wei Jihui 08 Feb '22

08 Feb '22

On Mon, Feb 07, 2022 at 04:02:52PM +0800, Li Ping wrote: >[why] >if no parameter is transferred, the default value is not set, As a >result, data cannot be obtained. > 我已经撤销了这个patch,合入最新的patch。 Thanks, Jihui >Signed-off-by: Li Ping <1477412247(a)qq.com> >--- > src/lib/web_backend.rb | 4 +++- > 1 file changed, 3 insertions(+), 1 deletion(-) > >diff --git a/src/lib/web_backend.rb b/src/lib/web_backend.rb >index f47950e6..571d7c64 100644 >--- a/src/lib/web_backend.rb >+++ b/src/lib/web_backend.rb >@@ -1026,8 +1026,10 @@ end > def get_srpm_info(params) > begin > my_data = MyData.new >+ page_size = params['page_size'] || 10 >+ page_num = params['page_num'] || 0 > info=[] >- srpm_infos = my_data.get_srpm_info(size: params['page_size'].to_i, from: params['page_num'].to_i) >+ srpm_infos = my_data.get_srpm_info(size: page_size, from: page_num) > srpm_infos['hits']['hits'].each do |source| > info << source['_source'] > end >-- >2.23.0 >

1 0

Re: [PATCH lkp-tests 3/3] lkp-tests/*: add new flow for install lkp depends
by Wei Jihui 08 Feb '22

08 Feb '22

On Sun, Jan 30, 2022 at 08:58:23AM +0800, Wang Yong wrote: >[why] >install lkp depends before test suite to satisfy lkp-tests > >Signed-off-by: Wang Yong <wangyong187(a)huawei.com> >--- > lib/job2sh.rb | 2 ++ > setup/install_depends | 49 +++++++++++++++++++++++++++++++++++-------- > 2 files changed, 42 insertions(+), 9 deletions(-) > >diff --git a/lib/job2sh.rb b/lib/job2sh.rb >index 4e8bf97fb..32caf2ade 100755 >--- a/lib/job2sh.rb >+++ b/lib/job2sh.rb >@@ -316,6 +316,7 @@ class Job2sh < Job > @monitors = available_programs(:monitors) > @setups = available_programs(:setup) > @programs = available_programs(:workload_elements) >+ @lkp = {"install_depends"=>{}} how about use '@install_depends'? Thanks, Jihui > > unless job > job = (@jobx || @job).clone # a shallow copy so that delete_if won't impact @job >@@ -336,6 +337,7 @@ class Job2sh < Job > out_line > out_line "\texport_top_env" > out_line >+ parse_hash [], @lkp > parse_hash [], job > out_line "}\n\n" > @script_lines >diff --git a/setup/install_depends b/setup/install_depends >index b5390cad1..8a6da808e 100755 >--- a/setup/install_depends >+++ b/setup/install_depends >@@ -1,6 +1,7 @@ > #!/bin/sh > > . $LKP_SRC/lib/detect-system.sh >+. $LKP_SRC/lib/install.sh > > detect_system > >@@ -16,8 +17,6 @@ set_ubuntu_debian() > -e "s|http://archive.ubuntu.com|${mirror_addr}|g" \ > -i.bak \ > /etc/apt/sources.list >- >- apt-get install -yqm "${packages}" > } > > set_suse() >@@ -25,8 +24,6 @@ set_suse() > sed -e "s|^baseurl=http://download.opensuse.org|baseurl=${mirror_addr}/opensuse|g" \ > -i.bak \ > /etc/zypp/repos.d/repo-*.repo >- >- zypper -q install -y "${packages}" > } > > set_archlinux() >@@ -36,8 +33,6 @@ set_archlinux() > else > sed -i "1s;^;Server\\ =\\ ${mirror_addr}/archlinux/\$arch/\$repo\\n;" /etc/pacman.d/mirrorlist > fi >- >- pacman -Syq --noconfirm --needed "${packages}" > } > > set_fedora() >@@ -46,8 +41,6 @@ set_fedora() > -e "s|^#baseurl=http://download.example/pub/fedora/linux|baseurl=${mirror_addr}/fedora|g" \ > -i.bak \ > /etc/yum.repos.d/fedora*.repo >- >- dnf install -y -q "${packages}" > } > > set_centos() >@@ -74,7 +67,43 @@ set_centos() > ;; > > esac >- yum install -y -q "${packages}" >+} >+ >+adapt_lkp_depends() >+{ >+ os=$(echo "${_system_name}" | tr '[:upper:]' '[:lower:]') >+ lkp_depends=$(get_dependency_packages $os lkp | xargs) >+} >+ >+get_package_manager() >+{ >+ has_cmd "yum" && installer="yum" >+ has_cmd "dnf" && installer="dnf" && return >+ has_cmd "apt-get" && installer="apt-get" && return >+ has_cmd "pacman" && installer="pacman" && return >+ has_cmd "zypper" && installer="zypper" && return >+} >+ >+install_depends() >+{ >+ packages="$1" >+ >+ get_package_manager >+ case "$installer" in >+ apt-get) >+ export DEBIAN_FRONTEND=noninteractive >+ "$installer" install -yqm ${packages} >+ ;; >+ dnf|yum) >+ "$installer" install -y -q --skip-broken ${packages} >+ ;; >+ pacman) >+ "$installer" -Sy --noconfirm --needed ${packages} >+ ;; >+ zypper) >+ "$installer" -q install -y ${packages} >+ ;; >+ esac > } > > set_local_mirror() >@@ -104,3 +133,5 @@ set_local_mirror() > } > > set_local_mirror >+adapt_lkp_depends >+install_depends "${lkp_depends}" >-- >2.23.0 >

1 0