Integrating IB with Github Actions

Hi there,

We wanted to see how easy it would be to integrate IB with Github Actions and behold…thanks to the great work of our DevOps expert Ori Hoch - we got it to work!
Check out his post below…

Would love to hear if anyone else out there would benefit from such an integration, tried it this way or had success doing it another way.

Which other CI tools would you see yourselves using with IB?

DISCLAIMER: this is not an official supported release

Now, the post…

The final result will be a fully automated continuous integration flow which will build the project on every repository push event. This workflow has minimal dependencies on specific GitHub Actions or Incredibuild functionality and could be adapted for any build system.

Prerequisites

In order to follow the instructions you should have set-up the Incredibuild system on AWS

If you already have a build system set up whether it’s Incredibuild or another build system, you can still follow-along but will have to make the appropriate changes.

High-level architecture overview

The CI workflow will be triggered from a push event to the project’s repository

GitHub Actions will handle the event and trigger the build job on the Incredibuild build system via the SSH service on the Incredibuild-coordinator server. This will allow secure, encrypted access.

The build script will download the relevant code from GitHub and run the build. All the code and the build output will be available on the Incredibuild-coordinator server.

Create an SSH key and allow access to the Incredibuild-coordinator server

We will use SSH to secure the connection between GitHub and the Incredibuild-coordinator server.

Most Linux distributions should have the required command to create an SSH key, if you find that you don’t, you can connect to the Incredibuild-coordinator server and run the command from there.

Create the SSH key using the following command:


ssh-keygen -t ed25519 -C "incredibuild-github-actions" -f incredibuild-github-actions-ssh-key -N ""

-t ed25519 : This is the type of key pair we will create

-C “incredibuild-github-actions” : A comment to identify the public key

-f incredibuild-github-actions-ssh-key : The filename for the private key. The public key will be stored using the same name with suffix .pub.

-N “” : Key password - we keep it empty because we will use this in a non-interactive session

2 files should be created:

incredibuild-github-actions-ssh-key - the private key which the GitHub actions script will use to authenticate with the SSH service on incredibuild-coordinator server

incredibuild-github-actions-ssh-key.pub - the public key which the SSH service on incredibuild-coordinator server will use to authenticate the validity of the private key

Connect to the incredibuild-coordinator server (you can right-click the instance in AWS EC2 console and choose “connect”) and run the following command to allow the key to authenticate. Replace PUB_KEY_CONTENTS with the contents of the incredibuild-github-actions-ssh-key.pub file:


echo "PUB_KEY_CONTENTS" >> ~/.ssh/authorized_keys

Prepare the GitHub repository

We will use the opencv project which we used in the previous post.

Fork the repository to your GitHub account by clicking the “fork” button on the opencv repository main page.

The GitHub Actions workflow we will set up will require some secret values which we will add to the repository secrets.

In your forked repository - click the “Settings” button. On the left side-bar click on “Secrets” and click on the “New Repository Secret” button to add the following secrets:

Name: “IB_SSH_IP”, Value: Your incredibuild-coordinator server public IP

Name: “IB_SSH_PRIVATE_KEY”, Value: The contents of the private SSH key file you created in previous step (filename: incredibuild-github-actions-ssh-key)

Finally, enable GitHub Actions by clicking “Actions” in the settings side-bar, select “Allow All Actions” and click “Save”.

Create the GitHub Actions workflow

First, make sure there are available agents as once we commit the workflow a build will start. In AWS Console EC2 - click on “Auto Scaling Groups” and increase the “Incredibuild-agent” group capacity to 3 instances.
Github actions blog image B

In your forked opencv repository (from the GitHub web UI) - browse to directory .github/workflows, click on “Add file” → “Create new file”

Set the file name to “incredibuild.yml” and paste the following contents:


# The name of the workflow, displayed on GitHub Actions UI

name: trigger Incredibuild

# This workflow will be triggered on every Git push event to your repository

on: push

# Every workflow can have multiple jobs which will all run in parallel by default

jobs:

# The name of our build job, displayed on GitHub Actions UI

build:

# The OS image to use for this job

runs-on: ubuntu-18.04

# Every job has a series of steps which run sequentially. If any step fails the entire job stops and is marked as a failure.

steps:

# This is the script which runs on this step. The pipe character ("|") is used to start a yaml multiline string which contains our build script

- run: |

# Get the private key from the repository secrets and store it in a local file.

echo "${{ secrets.IB_SSH_PRIVATE_KEY }}" > ib_ssh_private_key &&\

# Restrict the file permissions, this is required by the ssh client.

chmod 400 ib_ssh_private_key &&\

# We specify a unique name for the build directory which will be used inside the incredibuild-coordinator server.

# This allows us to retrieve the build when done and to debug errors.

BUILD_DIR=opencv-${GITHUB_SHA}-`date +%Y%m%d%H%M%S` &&\

# Start a non-interactive SSH session to the incredibuild-coordinator server

ssh -i `pwd`/ib_ssh_private_key -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no ubuntu@${{ secrets.IB_SSH_IP }} "

echo BUILD_DIR: $BUILD_DIR &&\

mkdir $BUILD_DIR && cd $BUILD_DIR &&\

# Download the relevant code. GITHUB_REPOSITORY and GITHUB_SHA are provided by the GitHub Actions build system and

# contain the repository name and relevant commit sha for current build

curl -Lso opencv.zip https://github.com/${GITHUB_REPOSITORY}/archive/${GITHUB_SHA}.zip &&\

unzip opencv.zip && cd opencv-${GITHUB_SHA} &&\

# standard build script, same one we used in previous blog post to build the opencv project

# this will initiate 20 parallel processes distributed via the incredibuild-coordinator to the agents

mkdir build && cd build &&\

cmake .. &&\

ib_console make -j 20 &&\

echo Great Success! &&\

echo BUILD_DIR: $BUILD_DIR

"

This workflow starts an SSH session on the Incredibuild-coordinator server and uses it to Get the relevant code and initiate the build. Review the comments in the file for all the details.

Commit the change to your “main” branch and the action will immediately be triggered. You can see the running action under the “Actions” tab in your GitHub repository page.
Github actions blog image C

You can see the action which ran on this repository by following this URL.

Now you can connect to the Incredibuild-coordinator server and get the build output from the build directory (using the BUILD_DIR value which appears in the action output).

Summary

In this post we saw how to integrate the Incredibuild system with GitHub actions to allow a CI workflow to initiate a distributed build of a large project. This example could easily be adapted to initiate via a different CI system - as we are using a simple Bash script and on the other side - it could trigger a different build tool, as it depends on a standard SSH service as the interface between the parts.

4 Likes

Thanks Yohai! Very interesting :pray:

1 Like