[Compass-ci] Re: [TODO] provide LKP dependencies in container with busybox+curl

- use debian OS as host OS - make sure host installed busybox-static package and downloaded static-curl - mount busybox/curl into docker - create busybox symlinks in docker

mkdir /opt/busybox/bin/ cd /opt/busybox/bin/ for c in wget cpio do ln -s /path/to/busybox $c done PATH=$PATH:/opt/busybox/bin/

by fengguang

thanks

- download .cgz depends inside docker in LKP framework - extract .cgz

On Mon, Feb 07, 2022 at 02:26:54PM +0800, Wu Fengguang wrote:

Currently /c/compass-ci/providers/docker/run.sh has extra dependencies that should be simplified.

TODO1: install busybox in the host (I see no script to make sure it's installed in the host). If the host OS does not has the busybox package, it's fine to download pre-built busybox-static binary

TODO2: make sure busybox symlinks are created in a dir like /opt/busybox/bin/ inside the container, and _append_ it to PATH.

TODO3: install curl-static in the host and mount into container /opt/curl/bin/

   https://github.com/moparisthebest/static-curl

   Grab the latest release from one of these links, by CPU architecture:
       curl-amd64
       curl-i386
       curl-aarch64
       curl-armv7
       curl-ppc64le

busybox+curl should provide all the *required* dependencies for running LKP framework.

Thanks, Fengguang

On Tue, Jan 04, 2022 at 10:07:08AM +0800, Wu Fengguang wrote:

...

Let's cut down dependencies in /c/lkp-tests/distro/depends/lkp

For the depends added by

   commit 54e515a10156080bfabc8d880b3f704b7c29a702
   Author:     Wei Jihui <weijihuiall@163.com>
   AuthorDate: Mon Nov 8 09:59:15 2021 +0800
   Commit:     Wei Jihui <weijihuiall@163.com>
   CommitDate: Thu Nov 11 14:25:03 2021 +0800

       distro/depends/lkp: move cci-depends'depends to lkp's depends

we can use either

• busybox ftpput
• rsync
• lftp

for uploading files. Those packages are rarely depended by normal packages. In comparison, curl is depended by hundreds of other packages in Debian.

If use busybox, lkp can remove lots of dependencies.

However for remote testbox, its 'busybox ftpput' command may not work well

• need PASSIVE mode in order to go through firewall
• not secure over Internet
• some firewall may bann insecure service/ports
• not good to block testbox in slow/unreliable Internet connection

The same for rsync upload, if ssh is not used together.

But shall we upload all remote testbox results over Internet? I guess not -- it won't scale.

Here is a flexible upload model

testbox -> on job finish, tar/cpio all results into a single file (to avoid complex mirror function for ftpput) -> upload via any of ftp/rsync/https to a dedicated and temp server storage space (eg. /srv/result/upload/$jobid/ with md5sum verification) (proftpd: configure as upload only dir, disable read and list-dir) -> tell scheduler that the job is complete -> run a service to unpack a job's results into final result_root

remote testbox -> results uploaded to a local server storage -> local storage server runs a service to send results to central lab storage when asked by end user

Thanks, Fengguang