Collins exists to drive infrastructure automation.
Someone recently asked me to describe collins in a sentence. At Tumblr, it's the infrastructure source of truth and knowledge. Everything about Tumblr production environments is stored and encoded in Collins, and that data is used to drive all of our automation. Sometimes people refer to systems like this as a CMDB, or Configuration Management DataBase.
Collins started as a system to manage all of the physical servers, switches, racks, etc in Tumblr production environments. As we started to inventory hardware, IP addresses, software, and so on, we found the API and data gave us an excellent way to drive automation processes. Today Collins can do push button cluster (HBase, Hadoop, web, etc) deployment, drive configuration generation when hardware cluster topologies change, drive infrastructure updates when software configuration changes, and help manage software deploys.
Because of the loosely coupled design of Collins, consistently applied conventions are a system requirement. This document serves as a guide to those conventions as well as the basic core concepts of the collins system. If you're just interested in the basic howto or screenshots, click here.
Collins consists of the core application, a set of client libraries, the collins-shell, and an automation toolchain. Nothing but collins core is required.
Collins provides an easy to use API on top of a simple data model. You can extend collins through callbacks, plugins, and light-weight collins scripts.
Collins is extremely dumb. It knows about assets, their meta-data and asset logs. You can think of collins as a key/value store where each asset has its own set of key/value pairs. There are no relationships between assets other than the ones you, through convention, derive. The API makes it trivial to create and manage the tags (meta-data, key/value pairs) associated with an asset, and to query based on those tags.
Collins is intentionally dumb. It worries about basic authentication, clean API interactions, and data persistence. If you start thinking, "Hey, I should build X into Collins", you probably shouldn't. Collins supports both a plugins architecture (for things that actually in some way change the behavior of collins) as well as a very usable API (including clients in Python, Ruby and Bash). Nearly everything you might want to do can be accomplished via the API and anything that can't is doable as a plugin.
By infrastructure engineers, for infrastructure engineers.
We looked at a dozen or so open source and commercial products before coding began. Having an API and flexible workflow were requirements. The intake support for our specific hardware intake process also needed to be accounted for. We also needed something that would support not just virtualized systems, but was more tailored towards physical hardware. We found nothing that met the requirements we had. Things we looked at included:
Some of these are strictly for IPAM, some handled parts of what collins does. None of these were good fits for the initial problems we were trying to solve.
Collins can sit on top of existing asset management systems (we have done this at Tumblr with both SoftLayer and Amazon) or be the infrastructure source of truth for your entire environment. I personally found openstack to be the closest fit for what we were looking for but the number of changes required to make it work for Tumblr were substantial.
Up and running in 5 minutes.
Collins has 1 software requirement, Java 1.6 (with the unlimited strength JCE).
Collins is developed using Java 1.6 and in the tests I did with Java 1.7 on OS X collins performed poorly. Typing
should display something like
java version "1.6.0_35"
If you would like your data to persist between restarts, you should also have MySQL 5.5 (5.1 may work but is untested).
The simplest way to get started playing with Collins is to use the Docker image. Make sure you have a functioning Docker install. Docker repository
docker run -p 9000:9000 tumblr/collins
scripts/setupand follow the instructions
Note that the above method will only work with MySQL. If you don't have MySQL available you can follow the instructions below which will use an in memory database.
cd ~/src/ & & git clone email@example.com:tumblr/collins.git
~/src/play-2.0.8/playand hit enter (this will take a minute the first time)
compileand hit enter, you should see no errors. We're just testing your environment, you normally won't do this.
runand then visit
http://localhost:9000in your browser. To run with a different configuration file (the default is
conf/application.conf), instead of typing run type
run -Dconfig.file=/path/to/alt.conf. Login with username
There is a supplied population script at
scripts/populate.sh which will create 10 assets in your local collins instance for you to play around with.
There is a supplied package script at
scripts/package.sh which will produce a fat zip with all the requirements and dependencies. This is what we deploy to production at Tumblr.
There is a supplied init script at
scripts/collins.sh. This is a useful starting point for running the app and is what we use to run the app in production at Tumblr. This script is also bundled by the
scripts/package.sh script. Note that the supplied init script requires daemon.
By default the authentication is provided via the
authentication.type=default). Usernames/passwords for this provider are below.
You can change the auth provider by setting
conf/application.conf. If you set
file, the default file used
conf/users.conf (which can be changed by setting
The usernames/passwords/groups in the default
users.conf file are below.
The script used to generate the file based passwords is available at
There is more complete authentication configuration documentation available here
If you want to help develop collins, you'll need to follow the instructions in "Software Setup" above. Then, to use your favorite IDE, follow the instructions on how to set it up using play. Setting up your preferred IDE with Play Framework
Vendorizing the Docker image
If you want to use the Docker build as the base for your production instance, you ought to vendorize the image with your own configurations. A good list of things you should override is listed below, along with a sample Dockerfile to produce your build. The Tumblr provided build uses the H2 in-memory database, development credentials, and lots of sample features turned on and configured. You will certainly want to tune the settings in the build (see https://github.com/tumblr/collins/tree/master/conf/docker for all the included configs).
Here is an example Dockerfile that will vendorize the Tumblr build with your own configs that live in ./conf/vendor/
FROM tumblr/collins ADD ./conf/vendor/production.conf /opt/collins/conf/production.conf ADD ./conf/vendor/database.conf /opt/collins/conf/database.conf ADD ./conf/vendor/validations.conf /opt/collins/conf/validations.conf ADD ./conf/vendor/authentication.conf /opt/collins/conf/authentication.conf ADD ./conf/vendor/users.conf /opt/collins/conf/users.conf ADD ./conf/vendor/profiles.yaml /opt/collins/conf/profiles.yaml ADD ./conf/vendor/permissions.yaml /opt/collins/conf/permissions.yaml
Then you can
docker build -t omnicorpllc/collins:123 . and
docker run -p 9000:9000 omnicorpllc/collins:123
A number of people were responsible for Collins growing into the useful tool that it has.