I’ve just finished configuring two different projects to use Gitlab CI/CD workflows on our v14.8 self-hosted instance, and a lot of the detail on the web is a little out of date, so here’s my overview of doing two slightly different workflows for two different kinds of project.

stages: based workflow

The first is a for a website that deploys to staging whenever it’s pushed. I configured this first, so this uses the named stages workflow that a lot of the documentation talks about.

There are 3 stages:

stages:
  - build
  - test
  - deploy

All the jobs that follow reference the stage: they belong to:

build-js:
  stage: build
  script:
    - yarn install

So this build-js job belongs to the build stage. There’s another job build-environment in in the build stage:

build-environment:
  variables:
    GIT_STRATEGY: none
  stage: build
  script:
    - ./script/ci-cd/create-git-ignored-needed-files

This job deliberately doesn’t do another git clone - if it did, we’d lose the yarn install files from the previous job. I should probably combine these two jobs into one:

build:
  stage: build
  script:
    - yarn install
    - ./script/ci-cd/create-git-ignored-needed-files

The next job belongs to the test stage, again, we don’t want another fresh git clone to happen, or we’ll lose the built js and other files that the build stage made, so that the tests can run with all the files they need:

integration-test:
  variables:
    GIT_STRATEGY: none
  stage: test
  script:
    - RAILS_ENV=$RAILS_ENV rake dbtest:integration

And then there’s the staging:deploy job in the deploy stage. The job names aren’t important - they just need to reference the correct stage if you’re doing stages: based workflows:

staging:deploy:
  stage: deploy
  script:
    - script/gitlab-deploy-staging

needs: based stageless workflow

Since Gitlab v14.2 you’ve been able to use stageless pipelines. You use a needs: keyword to reference the previous job that must have completed successfully before that job can run. If you’re familiar with Makefile recipe dependencies and prerequsites you’ll feel at home.

I used this style of workflow for the second project, which is a cross-platform Android, iOS, macos and Windows app, using Fastlane under the hood, to manage much of the building, testing and releases.

gitlab-runner on macos

I had one gotcha with the gitlab-runner running on a mac mini that’s building the macos and ios releases - you have to enable autologin for the account the gitlab-runner is run under. This is mentioned in the gitlab documentation, but it’s easy to miss:

Currently, the only proven way for it to work in macOS is by running the service in user-mode. Since the service will be running only when the user is logged in, you should enable auto-login on your macOS machine. The service will be launched as a LaunchAgent. By using LaunchAgents, the builds will be able to perform UI interactions, making it possible to run and test in the iOS simulator.

I was ssh-ing into the mac mini after a reboot and trying to see why the workflow was tagged with stuck and the runner wasn’t picking up jobs.

I tried to the start the process:

gitlab-runner start

and was just seeing:

Runtime platform    arch=amd64 os=darwin pid=7827 revision=bd40e3da version=14.9.1
FATAL: Failed to start gitlab-runner: exit status 134

because I’d skipped over the instructions to enable auto-login on the macOS machine.

You can add the gitlab-runner as a system LaunchDaemon (the gitlab-runner.plist will live under /Library/LaunchDaemons), but it won’t be able to to run and test in the iOS simulator.

It’s worth noting that macOS also has LaunchDaemons, services running completely in background. LaunchDaemons are run on system startup, but they don’t have the same access to UI interactions as LaunchAgents. You can try to run the Runner’s service as a LaunchDaemon, but this mode of operation is not currently supported.

You also want to make sure that the logged-in user doesn’t sleep - typically the mac’s going to be a headless server or just not connected to a monitor, and you don’t want it snoozing:

sudo pmset -a sleep 0; sudo pmset -a hibernatemode 0; sudo pmset -a disablesleep 1

Here’s the .gitlab-ci.yml file, with a load of supporting comments to explain what’s going on.

workflow:
  rules:
    # We only want to run the workflow if we're on the "feature/ci-test" branch
    # for now
    - if: '$CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH == "feature/ci-test"'
    - when: never # don't run for any other branches

# We're going to use a stageless pipeline as outlined in
# https://about.gitlab.com/releases/2021/08/22/gitlab-14-2-released/#stageless-pipelines
# but it's worth noting that there is still an implicit stage which we
# hang our jobs off, because none are explicitly defined, these are the defaults:
#
# stages:
#   - build
#   - test
#   - deploy
#
# and also, the first defined job, will implictly belong to the `build` stage
# and then the rest of the jobs hang of this via `needs:`
#
# The other thing that's not made very clear in the gitlab docs, is that each
# job will *re-checkout the project from git* so if your jobs rely on build
# artifacts from a previous stage, then you need to prevent the git reset
# from happening for that job with
# variables:
#   GIT_STRATEGY: none
#
######################### Checkout the project #################################
checkout:
  script:
    # These are just bash shell script calls..
    - echo "Checking out"

############################### Build jobs #####################################

build-android:
  needs: ["checkout"]
  variables:
    GIT_STRATEGY: none # don't recheckout the project
  artifacts:
    paths:
      - binaries/android
  script: ./script/build-android

build-ios:
  needs: ["checkout"]
  variables:
    GIT_STRATEGY: none # don't recheckout the project
  artifacts:
    paths:
      - binaries/ios
  script: ./script/build-ios

build-macos:
  needs: ["checkout"]
  variables:
    GIT_STRATEGY: none # don't recheckout the project
  artifacts:
    paths:
      - binaries/macos
  script: ./script/build-macos

build-windows:
  needs: ["checkout"]
  variables:
    GIT_STRATEGY: none # don't recheckout the project
  artifacts:
    paths:
      - binaries/windows
  script: ./script/build-windows

################################ Test jobs #####################################

android-tests:
  needs: ["build-android"] # will run as soon as the build-android job completes
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/test-android

ios-tests:
  needs: ["build-ios"]
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/test-ios

macos-tests:
  needs: ["build-macos"]
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/test-macos

windows-tests:
  needs: ["build-windows"]
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/test-windows

################################ Release jobs ##################################
#
# These *shouldn't* happen each time we push a build. We might just be testing
# a new feature, or doing some intermediate work prior to doing a full
# release, and the app store releases should be triggered only when we're ready.
# Because of this, we've set a manual rule, so that someone needs to click a
# button in the Gitlab jobs UI to action the relevant release job.
################################################################################
android-release:
  rules:
    - when: manual
      allow_failure: true  # It's a manual job, so we don't want the entire
                           # pipeline to fail just because this job didn't run
  needs: ["android-tests"] # This can be run once android-tests is successful
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/google-play-store-release

ios-release:
  rules:
    - when: manual
      allow_failure: true
  needs: ["ios-tests"]
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/ios-app-store-release

macos-release:
  rules:
    - when: manual
      allow_failure: true
  needs: ["macos-tests"]
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/macos-app-store-release

windows-release:
  rules:
    - when: manual
      allow_failure: true
  needs: ["windows-tests"]
  variables:
    GIT_STRATEGY: none
  script:
    - ./script/window-app-store-release