run kitchen tests on travis

[ofir-petrushka]

This will make all current kitchen test run in travis in parallel (while still running rubocop and foodcritic)

@chr4 I know you mentioned wanting to see a way for this to work.

This will be red in travis as there are things to fix for some of the suites and/or platforms and/or combo of which. (So merging red in hopes to fix or differing merge until can fix all issues is up to you..)
I will not currently have the time to continue hacking on this in the short term, but I hope this will be useful even if just to trigger @jeffbyrnes to do it in a better way ;).

What we would have wanted to see is a more dynamic way for travis to do the platform X kitchen suite, but it seems this isn't currently supported.
The only way to keep the list dynamically is not to use Travis parallel execution which would result in much longer run times. (current ones are quite long as well for the free usage tier which seems to run a max of 5 jobs at the time of test).

@jeffbyrnes I'd admit a lot of the code, including this, uses Gems and Rakefile because that's what I found to re-use ;). (path of least resistance)
This is based on zuazo/kitchen-in-travis as you can easily see in the travis file.

[chr4]

Doesn't have sudo = true a huge impact on performance? As far as I remember, it then can't be run in containers, but instead virtual machines.

[ofir-petrushka]

Travis has 3 run environments:

• sudo = false is running inside containers on their side, the big bonus is the startup time. The issues are with limited actions allowed to run. As the name suggest, can't run anything that need elevated privileges. Can't run "sudo" commands at all.
• sudo = true runs on the old vm's style, it allows you to run sudo commands, basically allowing you to install anything and run anything.
• (There is another less known vm mode that was alpha last time I checked, used to run docker stuff on it. This need enabling from Travis side per repo, need to ask by tickets... I think this is the current status https://docs.travis-ci.com/user/trusty-ci-environment/)

The docker Travis mode is very limited and you can't do a lot of things.
I don't remember the exact limits you either can't run docker containers inside it at all or you are very limited to some really tiny amount of use cases.

Travis page on using docker said use "sudo: required"
https://docs.travis-ci.com/user/docker/

Basically the penalty is in startup time for the instance and I think you get a bit of a different node spec to run on.
But they are all still fully supported by Travis and aren't going to be deprecated anytime soon as there is no replacement for some of the use cases.

[jeffbyrnes]

What if you rewrote this like so:

env:
  matrix:
    - SUITE='generic-agent-default-centos-72'
    - SUITE='php-agent-default-centos-72'
    - SUITE='php-external-default-centos-72'

script:
- rake rubocop
- rake foodcritic .
- bundle exec kitchen test $SUITE

DRY things up a bit.

Also as I mentioned you can just install ChefDK using APT, and then your script could be:

script:
- cookstyle
- foodcritic .
- kitchen test $SUITE

Best part of cookstyle is that it uses a slower-moving, Chef-centric set of Rubocop rules, so your tests won’t break just because Rubocop was updated & has new rules to follow.

[ofir-petrushka]

I am inclined to avoid coupling different test types inside the same CI task as it makes them harder to use IMHO.

If you want to slow down package updates, you can be more strict with versions inside the Gemfile and/or include the .lock file into the repo.
http://bundler.io/

If your concerned about the starting prefix of TASK, we can probably introduce some Rakfile magic..
http://stackoverflow.com/questions/9539324/generate-rake-test-tasks-dynamically-based-on-existing-test-files-in-a-rakefil
I know you prefer to move away from it...

[jeffbyrnes]

@ofir-petrushka ahh, I see. Yeah, makes total sense to have the RuboCop & Foodcritic linting run as their own sub-builds. Worth the lack of DRY-ness.

As for “slow down package updates”, it’s not about that. The whole point of ChefDK is to provide a sensible, compatible environment as a package appropriate for developing using Chef. If you take a look at ChefCo’s own cookbooks, this is how they test, which I take as a strong recommendation: sous-chefs/mysql@b59be78

[jeffbyrnes]

The third run environment is the Trusty beta (i.e., Ubuntu 14.04).

Using the non-containerized environment has a startup penalty, but does let you use Docker, which does let you do the Kitchen runs in Travis without needing to use a cloud provider.

[jeffbyrnes]

Going through old issues & PRs, and this is an absolutely ancient PR, and probably should simply be closed at this point.