Collins

Infrastructure management for engineers

Collins exists to drive infrastructure automation.

Someone recently asked me to describe collins in a sentence. At Tumblr, it's the infrastructure source of truth and knowledge. Everything about Tumblr production environments is stored and encoded in Collins, and that data is used to drive all of our automation. Sometimes people refer to systems like this as a CMDB, or Configuration Management DataBase.

Collins started as a system to manage all of the physical servers, switches, racks, etc in Tumblr production environments. As we started to inventory hardware, IP addresses, software, and so on, we found the API and data gave us an excellent way to drive automation processes. Today Collins can do push button cluster (HBase, Hadoop, web, etc) deployment, drive configuration generation when hardware cluster topologies change, drive infrastructure updates when software configuration changes, and help manage software deploys.

Because of the loosely coupled design of Collins, consistently applied conventions are a system requirement. This document serves as a guide to those conventions as well as the basic core concepts of the collins system. If you're just interested in the basic howto or screenshots, click here.

Collins consists of the core application, a set of client libraries, the collins-shell, and an automation toolchain. Nothing but collins core is required.

  • Documented, consistent REST API
  • Client API libraries in Ruby, Python and Bash
  • A collins command line shell for scripting and automation
  • A callback system for hooking into asset lifecycle events
  • Multiple authentication backends (LDAP, files, etc)
  • Per-feature and per-endpoint configurable permissions
  • Flexible Web and API based provisioning (PXE, SSH/grub/kickstart, API (e.g. AWS))
  • Power management support
  • IP Address allocation and management support
  • Support for running collins in distributed mode across multiple datacenters
  • Flexible data-model supporting multi-dimensional keys/values per asset
  • Simple key/value queries as well as a more advanced query language
  • A script engine for modifying collins behavior
  • Internationalized language support

Collins provides an easy to use API on top of a simple data model. You can extend collins through callbacks, plugins, and light-weight collins scripts.

Collins is extremely dumb. It knows about assets, their meta-data and asset logs. You can think of collins as a key/value store where each asset has its own set of key/value pairs. There are no relationships between assets other than the ones you, through convention, derive. The API makes it trivial to create and manage the tags (meta-data, key/value pairs) associated with an asset, and to query based on those tags.

Collins is intentionally dumb. It worries about basic authentication, clean API interactions, and data persistence. If you start thinking, "Hey, I should build X into Collins", you probably shouldn't. Collins supports both a plugins architecture (for things that actually in some way change the behavior of collins) as well as a very usable API (including clients in Python, Ruby and Bash). Nearly everything you might want to do can be accomplished via the API and anything that can't is doable as a plugin.

By infrastructure engineers, for infrastructure engineers.

We looked at a dozen or so open source and commercial products before coding began. Having an API and flexible workflow were requirements. The intake support for our specific hardware intake process also needed to be accounted for. We also needed something that would support not just virtualized systems, but was more tailored towards physical hardware. We found nothing that met the requirements we had. Things we looked at included:

Some of these are strictly for IPAM, some handled parts of what collins does. None of these were good fits for the initial problems we were trying to solve.

Collins can sit on top of existing asset management systems (we have done this at Tumblr with both SoftLayer and Amazon) or be the infrastructure source of truth for your entire environment. I personally found openstack to be the closest fit for what we were looking for but the number of changes required to make it work for Tumblr were substantial.

Up and running in 5 minutes.

Software Requirements

Collins has 1 software requirement, Java 1.6 (with the unlimited strength JCE). Collins is developed using Java 1.6 and in the tests I did with Java 1.7 on OS X collins performed poorly. Typing java -version should display something like java version "1.6.0_35"
If you would like your data to persist between restarts, you should also have MySQL 5.5 (5.1 may work but is untested).

Ultra Quick

  1. Download the latest collins zip from here
  2. Unpack the zip file
  3. run scripts/setup and follow the instructions

Note that the above method will only work with MySQL. If you don't have MySQL available you can follow the instructions below which will use an in memory database.

Software Setup

  1. Download the Play Framework
  2. Unpack the zip into $HOME/src/play-2.0.8
  3. cd ~/src/ & & git clone git@github.com:tumblr/collins.git
  4. In ~/src/collins type ~/src/play-2.0.8/play and hit enter (this will take a minute the first time)
  5. Once you are in the console type compile and hit enter, you should see no errors. We're just testing your environment, you normally won't do this.
  6. Type run and then visit http://localhost:9000 in your browser. To run with a different configuration file (the default is conf/application.conf), instead of typing run type run -Dconfig.file=/path/to/alt.conf. Login with username blake and password admin:first.

There is a quickstart screencast up here. This has changed slightly since it was published, but essentially just walks you through the above an additionally installs and sets up the collins-shell.

Support Scripts

There is a supplied population script at scripts/populate.sh which will create 10 assets in your local collins instance for you to play around with.
There is a supplied package script at scripts/package.sh which will produce a fat zip with all the requirements and dependencies. This is what we deploy to production at Tumblr.
There is a supplied init script at scripts/collins.sh. This is a useful starting point for running the app and is what we use to run the app in production at Tumblr. This script is also bundled by the scripts/package.sh script. Note that the supplied init script requires daemon.

Authentication

By default the authentication is provided via the MockAuthenticationProvider (set in conf/application.conf as authentication.type=default). Usernames/passwords for this provider are below.

You can change the auth provider by setting authentication.type to file or ldap in conf/application.conf. If you set authentication.type to file, the default file used is conf/users.conf (which can be changed by setting authentication.file.userfile=/path/to/users.conf). The usernames/passwords/groups in the default users.conf file are below.

UsernamePasswordGroups
blake admin:first engineering,infra,ops
matt foobar engineering,management
test fizz none
joeengineer flah engineering
UsernamePasswordGroups
blake 8vCeb3h8F863 engineering,infra,ops
matt 0592607689Cr engineering,management
test h2624A7WJA3y none
joeengineer 4ZRl5311al77 engineering

The script used to generate the file based passwords is available at scripts/gen_passwords.sh

There is more complete authentication configuration documentation available here

Contributing to Collins

If you want to help develop collins, you'll need to follow the instructions in "Software Setup" above. Then, to use your favorite IDE, follow the instructions on how to set it up using play. Setting up your preferred IDE with Play Framework

A number of people were responsible for Collins growing into the useful tool that it has.

  • Dan Simon for being the only other collins contributor for months
  • Steve Salevan for jumping in at the last minute to help get collins into shape to open-source
  • Joshua Hoffman for working with me on the initial specs
  • Dallas Marlow for figuring out all the ways to use the API that I didn't anticipate
  • Brad McDuffie for providing feedback on collins-shell (and making it usable)
  • Everyone else who helped with testing or provided feedback