Continuous Integration in GitLab¶
Note
Some knowledge of the GitLab CI configuration is assumed
The gitlab-ci.yml
CI controller¶
It is proposed that we add pipelines as stages in the gitlab-ci, thereby utilising the test infrastructure to perform basic pipeline tests.
The CI file is organised in stages:
image: ubuntu:18.04
stages:
- build-all
- build-yanda
- test
- deploy
before_script:
- echo "Before script section"
- echo "For example you might run an update here or install a build dependency"
- echo "Or perhaps you might print out some debugging details"
after_script:
- echo "After script section"
- echo "For example you might do some cleanup here"
build-all:
stage: build-all
script:
- echo "Common build for everyone"
- mkdir ${CI_PROJECT_DIR}/products
artifacts:
paths:
- ${CI_PROJECT_DIR}/products
build-yanda:
stage: build-yanda
script:
- echo "Do your build here"
- mkdir ${CI_PROJECT_DIR}/products/yandasoft
artifacts:
paths:
- ${CI_PROJECT_DIR}/products/yandasoft
test-yandasoft:
image: registry.gitlab.com/askapsdp/all_yandasoft
stage: test
script:
- echo "The YANDASoft test"
- apt-get update -y
- apt-get install -y ca-certificates curl --no-install-recommends
- apt-get install -y default-jdk
- cd pipelines/yandasoft/nextflow
- curl -s https://get.nextflow.io | bash
- ./nextflow run ./functest.nf
- mv products/* ${CI_PROJECT_DIR}/products/yandasoft/
artifacts:
paths:
- ${CI_PROJECT_DIR}/products/yandasoft
test-rascil:
image:
name: $CAR_OCI_REGISTRY_HOST/rascil-full:0.3.0
entrypoint: [""]
stage: test
before_script:
- echo "The RASCIL test"
- apt-get update
- apt-get -y install default-jdk
- cd pipelines/rascil/nextflow
- curl -s https://get.nextflow.io | bash
script:
- ./nextflow run rascil_imager.nf
- mv products/* ${CI_PROJECT_DIR}/products/rascil/
artifacts:
paths:
- ${CI_PROJECT_DIR}/products/rascil
expire_in: 1 week
test-other:
stage: test
script:
- echo "Do another parallel test here"
- echo "For example run a lint test"
deploy1:
stage: deploy
script:
- echo "Do your deploy here"
# Create Gitlab CI badges from CI metrics
# https://developer.skatelescope.org/en/latest/tools/continuousintegration.html#automated-collection-of-ci-health-metrics-as-part-of-the-ci-pipeline
include:
- project: 'ska-telescope/templates-repository'
file: 'gitlab-ci/includes/post_step.yml'
You can see in this example current as of March 2021. I have included a test pipeline for the YANDASoft imager. The test is performed inside a nextflow pipeline - it is proposed that this CI is extended by other pipelines. It is further possible that more tests can be performed upon the products archived by the pipelines.
Note
The NextFlow method is detailed in the YANDASoft pipeline. But this not meant to be prescriptive. This is simply one way of presenting a containerised workflow