Skip to content

infrastructure jenkins

Jump to bottom
Brian Barrett edited this page Apr 12, 2017 · 9 revisions

OMPI Jenkins Setup

The Open MPI community runs a Jenkins build server (and fleet of build workers) to check both pull requests and new commits. The server runs on aws.open-mpi.org, with builders running at a number of institutions as well as AWS. In addition, Mellanox and IBM run their own Jenkins master, as there is some significant level of trust between master and slave. This page does not document how to use Jenkins, but instead is notes on how Jenkins is currently configured. Brian and Howard have been doing most of the configuration lately.

EC2 Builders

EC2 Builders run in the Open MPI Production account (same place aws.open-mpi.org runs), using the Jenkins EC2 plugin, which does auto scaling and instance stop/terminate when idle. The configuration is pretty straight forward, configured in the master configuration screen on Jenkins. aws.open-mpi.org has a role that allows all the permissions needed to modify EC2 instance state, meaning that we don't embed AWS creds in Jenkins, but instead use the Use EC2 instance profile to obtain credentials option.

There are a number of disadvantages to starting/stoping an instance (roughly equivalent to turning off the machine, then turning it back on later), so we terminate instances when idle. This means we also build a new instance every time we need new workers. Initially, we did this by having a long CloudInit script that installed all the packages, but this slowed new instance startup time and sometimes made Jenkins angry. So we now custom-build AMIs for all our test platforms and skip the CloudInit script.

Current AMIs

AMI Description Parent AMI
ami-17960577 Amazon Linux 17.03 ami-8ca83fec
ami-fd9c0f9d FreeBSD 11 ami-6926f809
ami-0e9b086e RHEL 7.3 ami-6f68cf0f
ami-569c0f36 SLES 12 ami-e4a30084
ami-68950608 Ubuntu 16.04 ami-a58d0dc5

Linux Instances

Follow the instructions in https://github.com/open-mpi/ompi-scripts/blob/master/jenkins/linux-customize-ami.sh to build a new Jenkins builder AMI. Then configure Jenkins as below:

Configuration:

Instance Type: T2Micro
Availability Zone: us-west-2c
Security group names: sg-f87d7a9e
Remote FS root: /home/<DEFAULT_USER>
Remote user: DEFAULT_USER
AMI Type: unix
Root command prefix: sudo
Remote ssh port: 22
Labels: <LABELS FROM LINUX CUSTOMIZE AMI>
Idle termination time: -5
User Date:
TBD
Number of Executors: 1
Stop/Disconnect on Idle Timeout: True
Subnet ID for VPC: subnet-a9e154f1
Instance Cap: 5
IAM Instance Profile: arn:aws:iam::518752846868:instance-profile/jenkins-worker
Associate Public IP: True

The Idle termination time: -5 is not a typo. EC2 bills in hour increments. So if an instance gets stopped after running 30 minutes or after 59 minutes, it's the same cost. -5 means Jenkins won't stop an instance until 5 minutes before the hour is up, maximizing instance reuse if there are multiple builds in the hour.

FreeBSD

To build a new AMI:

  • Create a t2.micro instance using the official RELEASE ami
  • run pkg install openjdk autoconf automake libtool gcc wget curl git as root on the AMI (default user: ec2-user)
  • Now's a good time to checkout / build Open MPI, just to verify everything's there:
  • Follow these instructions to create a new ami
    • Delete the file /root/.ssh/authorized_keys or /home/ec2-user/.ssh/authorized_keys (otherwise your SSH key will be able to log in to any instances launched from this AMI, and Jenkins won't be able to).
    • Create the file /firstboot on 10.0-RELEASE or /root/firstboot on earlier releases (otherwise the AMI you create won't know to download the new SSH public key for logins).
    • Stop the instance (but do not terminate it).
    • Use the ec2-create-image tool or the "Create Image" command in the EC2 Management Console to create an AMI from the instance.
    • Terminate the instance.

Configuration:

Instance Type: T2Micro
Availability Zone: us-west-2c
Security group names: sg-f87d7a9e
Remote FS root: /home/ec2-user
Remote user: ec2-user
AMI Type: unix
Root command prefix: su -m root -c
Remote ssh port: 22
Labels: ec2 freebsd freebsd_11.0
Idle termination time: -5
User Date:
TBD
Number of Executors: 1
Stop/Disconnect on Idle Timeout: True
Subnet ID for VPC: subnet-a9e154f1
Instance Cap: 5
IAM Instance Profile: arn:aws:iam::518752846868:instance-profile/jenkins-worker
Associate Public IP: True
Clone this wiki locally