- Compass-ci - mailweb.openeuler.org

Re: [PATCH v3 compass-ci] data-api: add es search api
by Wu Fengguang 11 Feb '22

11 Feb '22

On Wed, Feb 09, 2022 at 09:23:20PM +0800, weijihui wrote: >On Wed, Feb 09, 2022 at 01:21:14PM +0800, Wu Fengguang wrote: >>On Tue, Feb 08, 2022 at 03:10:57PM +0800, Wei Jihui wrote: >>>From: Wei Jihui <weijihuiall(a)163.com> >>> >>>1. /data_api/es/:index/es_search is es search by dsl >> >>是这样的吧： >> >> es_search => _search > >yes > >> >>>2. /data_api/es/:index/es_search_by_sql is es search by sql >> >>我看你后面用了 >> >> >+# curl -X POST localhost:10005/data_api/es/jobs/_search_by_sql >> >>然后调用的是 >> >> >+ @client.perform_request('GET', '_nlpcn/sql', {}, search_sql) >> >>但我们应该最终调用这个吧: >> >> https://github.com/opendistro-for-elasticsearch/sql >> >> POST https://<host>:<port>/_opendistro/_sql >> { >> "query": "SELECT * FROM my-index LIMIT 50" >> } >> >>相应的API是不是相应的写成 >> >> /data_api/_opendistro/_sql > >这个接口需要安装的插件没有我们用的es版本 >目前用的这个接口 >_nlpcn/sql 可以计划下升级到 _opendistro 和与之匹配的ES。 API层面，可以先用 _opendistro 免得折腾。 >> >>>-# curl -X POST localhost:10005/es_find -H 'Content-Type: application/json' -d '{ >>>+# curl -X POST localhost:10005/data_api/es/jobs/_search -H 'Content-Type: application/json' -d '{ >>> # "query": {"size":10, "sort": [{"start_time": {"order": "desc"}}]}, >>>-# "index": "jobs", >>>-# "my_token": "16132550-...", >>>-# "my_account": "auto-submit", >>>-# "query_type": "eql" # can be omitted in this scenario >>>+# "cci_authorization": { >> >>这个貌似不错： >> >> credentials > >cci_credentials? Yes. Thanks, Fengguang

1 0

Re: [PATCH todo 1/2] add todo for docker improvement
by Luan Shengde 11 Feb '22

11 Feb '22

pls ignore this mail Thanks, Luan Shengde On Fri, Feb 11, 2022 at 05:05:52PM +0800, Luan Shengde wrote: >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > .../todo/20220216-5d:docker-improvement.md | 35 +++++++++++++++++++ > 1 file changed, 35 insertions(+) > create mode 100644 people/LuanShengde/todo/20220216-5d:docker-improvement.md > >diff --git a/people/LuanShengde/todo/20220216-5d:docker-improvement.md b/people/LuanShengde/todo/20220216-5d:docker-improvement.md >new file mode 100644 >index 00000000..416c749d >--- /dev/null >+++ b/people/LuanShengde/todo/20220216-5d:docker-improvement.md >@@ -0,0 +1,35 @@ >+# docker improvement >+ >+## handle download failure and check dir besides /lkp and /opt >+ >+original: >+ curl_cmd: %x(curl -sS --create-dirs -o #{path}/#{name} #{url} && gzip -dc #{path}/#{name} | cpio -idu -D #{path}) >+ download cgz file and unpack it in one line >+ >+improvement: >+ curl_cmd: split download cgz file, check dir and unpack the cgz file into standalone lines. >+ system('wget -t 3 -T 20 -O #{path}/#{name} #{url} 2> /dev/null') >+ use system to call the linux command, it will retusn true/false >+ - true: execute success >+ - false: execute in failure >+ raise error if download in failure >+ >+ %x(gzip -dc path/cgz_file | cpio -idt) >+ use %x(command) can get the output of the command, extract the file list of the cgz file >+ check files from the file list and check if the file/dir is besides the /lkp or /opt >+ >+ if no files/dirs besides /lkp or /opt >+ %x(gzip -dc path/cgz_file | cpio -idu -D path) >+ >+ if exist files/dirs besides /lkp or /opt >+ save the cgz to opt >+ unpack the cgz file in container >+ >+ --- >+ maybe can ignore check the cgz file if it containers files/dirs besides /lkp or /opt >+ just unpack all the cgz file in the container >+ --- >+ >+## /usr/local/bin/busybox should be ro >+ >+## cancel mount unnecessary dir >-- >2.23.0 >

1 0

Re: [PATCH v2 compass-ci] assign-account-tool: update email content
by Luan Shengde 11 Feb '22

11 Feb '22

On Thu, Feb 10, 2022 at 09:53:48PM +0800, Wei Jihui wrote: >On Thu, Feb 10, 2022 at 03:58:43PM +0800, Luan Shengde wrote: >>From: Luan Shengde <shdluan(a)163.com> >> >>for only login with public key is allow now, update the email content. >> >>Signed-off-by: Luan Shengde <shdluan(a)163.com> >>--- >> .../assign-account/build-send-account-email.rb | 13 ++++++++++--- >> container/assign-account/build-update-email.rb | 13 +++++++++++-- >> 2 files changed, 21 insertions(+), 5 deletions(-) >> >>diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb >>index 3c414041..f30de78b 100755 >>--- a/container/assign-account/build-send-account-email.rb >>+++ b/container/assign-account/build-send-account-email.rb >>@@ -16,10 +16,17 @@ def build_message(email, account_info) >> Login command: >> ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} >> >>- Account password: >>- #{account_info['my_password']} >>+ You can directly use the public key you offered to login the account-ver server. >> >>- You can use your pub_key to login if you have offered one. >>+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >>+ >>+ name: Zhang Dewan >>+ phone: 18969058870 >>+ email: zhangdewan1(a)compass-ci.org >>+ >>+ name: Luan Shengde >>+ phone: 15109218229 >>+ email: luanshengde(a)compass-ci.org >> >> Suggest: >> If you use the password to login, change it in time. >>diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb >>index 40d6b3b3..0a298ac8 100755 >>--- a/container/assign-account/build-update-email.rb >>+++ b/container/assign-account/build-update-email.rb >>@@ -18,8 +18,17 @@ def build_update_message(email, account_info, conf_info) >> Login command: >> ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} >> >>- Account password: >>- #{account_info['my_password']} >>+ You can directly use the public key you offered to loging the jumper server. > > loging --> login? > > Thanks, > Jihui OK, I will fix it Thanks, Luan Shengde > > >>+ >>+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >>+ >>+ name: Zhang Dewan >>+ phone: 18969058870 >>+ email: zhangdewan1(a)compass-ci.org >>+ >>+ name: Luan Shengde >>+ phone: 15109218229 >>+ email: luanshengde(a)compass-ci.org >> >> regards >> compass-ci >>-- >>2.23.0 >>

1 0

Re: [PATCH v2 compass-ci] assign-account-tool: update email content
by weijihui 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 03:58:43PM +0800, Luan Shengde wrote: >From: Luan Shengde <shdluan(a)163.com> > >for only login with public key is allow now, update the email content. > >Signed-off-by: Luan Shengde <shdluan(a)163.com> >--- > .../assign-account/build-send-account-email.rb | 13 ++++++++++--- > container/assign-account/build-update-email.rb | 13 +++++++++++-- > 2 files changed, 21 insertions(+), 5 deletions(-) > >diff --git a/container/assign-account/build-send-account-email.rb b/container/assign-account/build-send-account-email.rb >index 3c414041..f30de78b 100755 >--- a/container/assign-account/build-send-account-email.rb >+++ b/container/assign-account/build-send-account-email.rb >@@ -16,10 +16,17 @@ def build_message(email, account_info) > Login command: > ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > >- Account password: >- #{account_info['my_password']} >+ You can directly use the public key you offered to login the account-ver server. > >- You can use your pub_key to login if you have offered one. >+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >+ >+ name: Zhang Dewan >+ phone: 18969058870 >+ email: zhangdewan1(a)compass-ci.org >+ >+ name: Luan Shengde >+ phone: 15109218229 >+ email: luanshengde(a)compass-ci.org > > Suggest: > If you use the password to login, change it in time. >diff --git a/container/assign-account/build-update-email.rb b/container/assign-account/build-update-email.rb >index 40d6b3b3..0a298ac8 100755 >--- a/container/assign-account/build-update-email.rb >+++ b/container/assign-account/build-update-email.rb >@@ -18,8 +18,17 @@ def build_update_message(email, account_info, conf_info) > Login command: > ssh -p #{account_info['jumper_port']} #{account_info['my_login_name']}@#{account_info['jumper_host']} > >- Account password: >- #{account_info['my_password']} >+ You can directly use the public key you offered to loging the jumper server. loging --> login? Thanks, Jihui >+ >+ Case your private key for the public key you offered has changed and lead to login failures, you can contact our technical support staff for help: >+ >+ name: Zhang Dewan >+ phone: 18969058870 >+ email: zhangdewan1(a)compass-ci.org >+ >+ name: Luan Shengde >+ phone: 15109218229 >+ email: luanshengde(a)compass-ci.org > > regards > compass-ci >-- >2.23.0 >

1 0

Re: reproducible-build TODO list
by weijihui 10 Feb '22

10 Feb '22

On Wed, Feb 09, 2022 at 10:35:20AM +0800, Lu Weitao wrote: >On Tue, Feb 08, 2022 at 08:20:37PM +0800, Wei Jihui wrote: >>On Mon, Feb 07, 2022 at 03:14:42PM +0800, Lu Weitao wrote: >>>## TODO list >>>- TODO1.1: script support set env for build env --- compass-ci jihui >> >>what should i do for this? >> > >user submit rpmbuild.yaml will with some build params like: >fs: >kernel: >reproducible_build_env: > hostname: > datetime: >rpmbuild: > new_param: > >we should make sure these params take effect in testbox no need reproducible_build_env, just write it like runtime. runtime: 1d hostname: cc datetime: bb Thanks, Jihui > > > >--------- >Thanks, >Lu Weitao > >- > >>Thanks, >>Jihui >> >> >>> >>>- TODO1.2: script record buildinfo file in testbox --- IT/compass-ci >>> - based on debian buildinfo >>> - TODO1.3: 输出用户指定的息into buildinfo? >>> new lib/buildinfo.sh --- IT >>> func1() >>> func2() >>> func3() >>> save to $result_root/buildinfo.yaml >>> >>> modify rpmbuild script to call the above funcs. --- compass-ci wangyong >>> >>>- TODO1.4: extract-stats servic 提取buildinfo --- compass-ci weitao >>> >>>- TODO2: 比较构建结果是否相同 --- compass-ci weitao >>> compass ci >>> >>> reuse data API es-find >>> compare in client side >>> >>> 格式和进一步优化: IT >>> client: ask use python? >>> >>>- TODO3: 基于已构建的job的buildinfo 重复构建 ---- IT >>> >>>- TODO4 toolset --- IT/compass-ci >>> option1: compass ci local 1-key deploy in 1VM (most simple for us) -- 暂不采纳 >>> option2: simple toolset run in native host, run only container, no VM -- 已采纳---IT >>> option3: submit self-made build container (test option2 regularly) to run remote in compass ci --compass-ci wangyong >>> >>> >>> >>>--------- >>>Thanks, >>>Lu Weitao >>>

1 0

Re: [TODO] draft 2022 targets
by Bai Jing 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 04:14:55PM +0800, Wu Fengguang wrote: >One more: > >Setup and run syzkaller, keep trigger kernel bugs and reproduce program >accumulate and reuse the above reproduce programs as kernel test case > >Thanks, >Fengguang > >On Wed, Feb 09, 2022 at 05:19:14PM +0800, Wu Fengguang wrote: >>众智项目候选 >>界面 todo owner >>- web interface luweitao >>- maintain dashboard zhangdewan >>加包 >>- OS package name mapping wuzhende >>- auto convert 3w archlinux packages wangyong >>- auto create 3w C/C++ spec liping >>- test 10w pypi upstream packages liyuanchao >>- rpm spec customize options from gentoo and ./configure output parsing >>nix >>- nix 1w upstream project tests >>- openeuler 3k overlay on nixpkgs >>- openEuler+Nix+musl docker images for popular apps, replacing alpine >>改进? >>- k8s integration luanshengde >>- selftest liuyinsi >>- bisect caoxueliang >> >>OS大赛项目候选 >>- YAML+python overlays merge tool >> >>Thanks, >>Fengguang >>

1 0

腾讯企业邮箱自动转发验证邮件
by luanshengde 10 Feb '22

10 Feb '22

亲爱的朋友：您好，腾讯企业邮箱用户 luanshengde(a)compass-ci.org 在邮箱中设置了将邮件转发至您的邮箱中。为避免不友善的转发给您带来骚扰，该设置需要您允许才能生效。如果您允许该转发设置，请点击以下链接：接受该转发请求   如果您点击上述链接无效，您可能处于纯文本模式下。请复制以下地址至浏览器地址栏进行访问： http://exmail.qq.com/cgi-bin/attrset?fun=autoreply&t=verifyfw_result&alias=…     腾讯企业邮箱腾讯企业邮箱门户：http://exmail.qq.com 企业邮箱帮助中心：http://service.exmail.qq.com

1 0

test mail for forward
by Cao Xueliang 10 Feb '22

10 Feb '22

test mail for forward

1 0

Re: [TODO] draft 2022 targets
by Bai Jing 10 Feb '22

10 Feb '22

On Thu, Feb 10, 2022 at 04:14:55PM +0800, Wu Fengguang wrote: >One more: > >Setup and run syzkaller, keep trigger kernel bugs and reproduce program >accumulate and reuse the above reproduce programs as kernel test case > >Thanks, >Fengguang > >On Wed, Feb 09, 2022 at 05:19:14PM +0800, Wu Fengguang wrote: >>众智项目候选 >>界面 todo owner >>- web interface luweitao >>- maintain dashboard zhangdewan >>加包 >>- OS package name mapping wuzhende >>- auto convert 3w archlinux packages wangyong >>- auto create 3w C/C++ spec liping >>- test 10w pypi upstream packages liyuanchao >>- rpm spec customize options from gentoo and ./configure output parsing >>nix >>- nix 1w upstream project tests >>- openeuler 3k overlay on nixpkgs >>- openEuler+Nix+musl docker images for popular apps, replacing alpine >>改进? >>- k8s integration luanshengde >>- selftest liuyinsi >>- bisect caoxueliang >> >>OS大赛项目候选 >>- YAML+python overlays merge tool >> >>Thanks, >>Fengguang >>

1 0

Re: extract buildinfo
by Lu Weitao 10 Feb '22

10 Feb '22

please help review this design --------- Thanks, Lu Weitao On Wed, Feb 09, 2022 at 05:20:12PM +0800, Lu Weitao wrote: ># extract buildinfo and save it to ES > todo from reproducible-build flow > > ``` > after build rpm, and upload buildinfo.yaml, rpmbuild to server > | > v > $result_root/rpmbuild.yaml ---> rpmbuild.json ---> merge to stats.json > $result_root/rpmbuild_info.yaml ---> rpmbuild_info.json ---> merge to stats.json > | > v > save stats.json to ES db # 为compare， diifoscope 提供数据源 > eg: > { > "id": xxx, > "os": "openeuler", > "arch": "aarch64" > "stats": { > "rpmbuild.hash_value": xxx, > ``` >## TODO1: improve compass-ci/sbin/result2stats, extract-stats service call it > support extract buildinfo > usage: result2stats $result_root > ``` > ... > $result_root/rpmbuild ---> rpmbuild.json ---> merge to stats.json > $result_root/buildinfo.yaml ---> buildinfo.json ---> merge to stats.json > ... > ``` >### input: result_root/buildinfo(buildinfo.yaml) >- e.g. samba_4.9.5+dfsg-5+deb10u2_arm64.buildinfo >``` >Format: 1.0 >Source: samba >Binary: samba samba-libs samba-common samba-common-bin smbclient samba-testsuite registry-tools samba-dev python-samba samba-dsdb-modules samba-vfs-modules libsmbclient libsmbclient-dev winbind libpam-winbind libnss-winbind libwbclient0 libwbclient-dev ctdb >Architecture: all arm64 source >Version: 2:4.9.5+dfsg-5+deb10u2 >Checksums-Md5: > 79ffac5466e2b36633e0f429644407d0 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > d523774b75cc4d31dce4696d41d68b8f 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Checksums-Sha1: > b8c10499c1b6b7c5111c7f2d8d617ed8a2ff938e 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > 1569bfdb4399f92852863f67153cce046a0cbb38 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Checksums-Sha256: > c9094fcc63930f665707c596134ea64e44d433d3882b3de544c2a83f574aafb9 3211 samba_4.9.5+dfsg-5+deb10u2.dsc > 3304c2fa4f7204682747bed70f7ce61b134b21f063f9ebab0aeb2e3af645b744 536500 winbind_4.9.5+dfsg-5+deb10u2_arm64.deb > ... >Build-Origin: Debian >Build-Architecture: arm64 >Build-Date: Sun, 30 Jan 2022 09:27:50 +0000 >Installed-Build-Depends: > adduser (= 3.118), > zlib1g-dev (= 1:1.2.11.dfsg-1) > ... >Environment: > DEB_BUILD_OPTIONS="parallel=128" > SOURCE_DATE_EPOCH="1638005690" >``` >### output result_root/buildinfo.json >``` >{ > "Hash": "SHA512", > "Format": 1.0, > "Build-Architecture": "i386", > "Source": "fweb (1.62-12)", > "Binary": "fweb fweb-doc", > "Architecture": "all i386", > "Version": "1.62-12+b2", > "Binary-Only-Changes": > "fweb (1.62-12+b2) sid; urgency=low, binary-only=yes > . > * Binary-only non-maintainer upload for amd64; no source changes. > * Rebuild for multiarch sync > . > -- amd64 / i386 Build Daemon (brahms) <buildd_amd64-brahms(a)buildd.debian.org> Wed, 30 May 2012 09:48:12 +0200 18:37:51 +0000", > ... >} >``` > > >## TODO2: save buildinfo to job(ES) > extract-service load $result_root/buildinfo.json > >- option1: merge job['buildinfo'](job['record']['buildinfo']) > meeting record: > job.record.build.xxx: v => possible solution option1 > 通用性好（job['record']['buildinfo'][xxx]） > 结构较复杂 > > job.info.build.xxx: v > 效果同上 > value type: > - output binary hash > - info (xxx.rpm) > > job['buildinfo'] > 结构简单，但只适用与构建任务，目前未发现其它类型的job有这种需求，如需使用buildinfo进行搜索，那这种方式更简单 > > **job like:** > ``` > { > "id": xxx, > "os": "openeuler", > ... > "buildinfo": { > "Hash": "SHA512", > "Format": 1.0, > ... > } > } > ``` > **advantage:** > 1. reuse job['buildinfo'] to submit again # we can also merge it to job.yaml > 2. 3rd tool such as: `diffoscope` get job['buildinfo'] > **disadvantage:** > 1. compare need get metric from buildinfo in addition > >- option2: merge buildinfo to job['stats'], job like: > ``` > { > "id": xxx, > "os": "openeuler", > "stats": { > "buildinfo.Hash": "SHA512", > "buildinfo.Format": 1.0, > ... > } > } > ``` > advantage / disadvantage: opposite of option1 > > job.stats.rpmbuild.xxx > job['stats']['rpmbuild.xxx'] > compare 不好区分buildinfo 和 rpmbuild > 需要变动lkp-tests/stats/rpmbuild，且buildinfo > result_root/rpmbuild > > job.stats.buildinfo.xxx => possible solution option2 > job['stats']['buildinfo.xxx'] > extract-stats 已经支持，无需改动代码 > 仅仅将buildinfo信息输出未 result_root/buildinfo.yaml > > compare 性能结果时忽略 buildinfo （可以新加perf-metric-ignore规则） > > job.stats.message.xxx / job.stats.xxx.message > 目前.message, .log 已经是perf-metric-ignore规则，可以考虑将buildinfo也加进去 > > > job.stats.info.build.buildinfo.xhhhhhxx => possible solution > 通用性更好，相当于对metric有个全面的分类 > 结构较多 > > > job.buildinfo.xxx（in option1） > >- option3: option1 + option2 > job like: > ``` > { > "id": xxx, > "os": "openeuler", > ... > "buildinfo": { > "Hash": "SHA512", > "Format": 1.0, > ... > }, > "stats": { > "buildinfo.Hash": "SHA512", > "buildinfo.Format": 1.0, > ... > } > } > > ``` > **advantage: opton1 + option2 ** > **disadvantage:** > 1. job bigger, --> save DB > 2. extract-service does one more thing >_______________________________________________ >Compass-ci mailing list -- compass-ci(a)openeuler.org >To unsubscribe send an email to compass-ci-leave(a)openeuler.org

1 0