Overview
The Collins tool chain is at your disposal
The Collins tool chain consists of a suite of tools, applications, and libraries you can use along with your collins installation. You technically don't have to use any of these, but hopefully as you'll see in the recipes guide, they're pretty damn useful.
These tools all share the same licensing as Collins core (Apache 2) and include only free and open source software. Most code tends to be Ruby based but when appropriate we have tried to either provide an SDK for other languages (as well as bash) or ensure that the code is user servicable. More details about these tools can be found throughout the rest of this document.
Sdk
API access from the language of your choice
We realize that you may want to access Collins from a variety of contexts and as such we provide an SDK for the languages and environments we use at Tumblr. While we don't have an SDK available for every language, the API is well documented so anyone inclined should have no problem implementing one.
The reference implementation is implemented with Ruby, and is considered the most complete SDK. Anyone looking for a sample implementation is encouraged to review the Ruby collins-client code. Anytime a new API endpoint is implemented, in addition to documentation, we include support in the Ruby collins client and if applicable in the collins shell.
The ruby gem is available via rubygems.org/gems/collins_client.
There is also a feature-complete API client written in Go. The source code is available at github.com/tumblr/go-collins. The full documentation can be found on GoDoc.
Additionally, many of the API endpoints support Bash friendly plain text output. This was previously used in projects that are now deprecated, so not much development is being done to support these Bash endpoints.
Finally, we have a very basic python client available, which started off only implementing the API code needed to support deploys at Tumblr. It is published to pypi
We welcome any community contributions to any of these libraries, as well as to the main collins codebase.
Configs
Manage multiple collins configs easily
Collins DC Helper
Configs to use collins tools are YAML files, which are generally stored in your home directory in ~/.collins.yml. The collins-cli gem allows you to create and manage these configurations. For example, to create a new config for a Collins instance in jfk01:
$ gem install collins-cli $ collins dc --new jfk01 --host https://collins.jfk01.company.com Enter username (default: gabe): Enter password: Now, select the Collins instance "jfk01" with 'collins dc jfk01' to access Collins $ cat ~/.collins.yml.jfk01 --- :host: https://collins.jfk01.company.com :username: gabe :password: 'foo' :timeout: 120 $ collins dc jfk01 $ ls -la ~/.collins.yml lrwxr-xr-x 1 gabe staff 31 Mar 15 13:44 /Users/gabe/.collins.yml -> /Users/gabe/.collins.yml.jfk01 $ collins dc -l jfk01 * ewr01 sfo01
If you have multiple Collins sites and need to frequently switch between collins configurations when using collins-cli, there is a handy helper to change between configs: collins dc jfk01. This will link ~/.collins.yml to point to ~/.collins.yml.jfk01. Now all future collins queries will use the jfk01 configuration.
You can also list your configured instances with collins dc -l.
NOTE: Any utility using the collins-auth gem uses a default search path for configs in order: ENV['COLLINS_CLIENT_CONFIG'], ~/.collins.yml, /etc/collins.yml, /var/db/collins.yml.
Bash Helper
WARNING: It is recommended to use collins dc as described above instead of this Bash function.
The following snippet of bash provides a limited subset of what collins dc does in the collins-cli gem. Add this to your .bashrc
collinsdc() {
if [[ -z $1 ]] ; then
l=$(basename $(readlink ~/.collins.yml))
echo "${l##.collins.yml.}"
else
if [[ ! -f ~/.collins.yml.$1 ]] ; then
echo "No collins config found for $1" >&2
return 2
fi
ln -sfn ~/.collins.yml.$1 ~/.collins.yml && echo "Using collins config for $1"
fi
}
You can keep multiple configs for each collins instance at ~/.collins.yml.<datacenter>, and switch between them easily. (Example is using collins-cli gem.)
$ ls .collins.yml.* .collins.yml.dc1 .collins.yml.dc2 .collins.yml.dc3 .collins.yml.local $ collinsdc dc1 $ collinsdc dc2 $ collins find -p FOO ...find assets in dc2...
Collins-cli
Simple CLI access to Collins
Overview
The Rubygem collins-cli allows you to interact with Collins from the command line. This includes changing attributes, provisioning machines, allocating IPs, maintenance-moding machines, etc. Download the gem from collins-cli on Rubygems.org. The source can be found at github.com/byxorna/collins-cli. This tool covers 98% of needs for administrators. For the remaining 2%, it is suggested to either use collins-shell, or write a quick ruby script.
Installation
$ gem install collins-cli
Setup
collins-cli uses Collins::Authenticator, so it looks for a configuration file in ~/.collins.yml. See configs to setup configurations.
Usage
Check out github.com/byxorna/collins-cli for documentation
$ collins Usage: collins <command> [options] Available commands: dc, datacenter: Manage multiple Collins configurations query, find: Search for assets in Collins modify, set: Add and remove attributes, change statuses, and log to assets log: Display log messages on assets provision: Provision assets power: Control and show power status state, status: Show and manage states and statuses via State API ipam, address, ipaddress: Allocate and delete IPs, show IP pools
Collins shell
CLI access to Collins and your infrastructure
Overview
WARNING: This utility is no longer maintained, and users should try to migrate over to collins-cli where possible.
One of the tools in the collins tool chain is the collins shell. The collins shell provides a command line tool with complete API access to all API endpoints. The collins shell also provides an interactive console, built around pry giving you a full ruby environment for task automation.
Installation
Note that rvm is not a requirement for using the collins shell. Ruby 1.9.x is however. I recommend using ruby 1.9.3 because that's what we develop with at Tumblr. 1.9.2 probably also works, 1.8.x will not work. There is additional information on installing rvm here.
$ curl -L get.rvm.io | bash -s stable --ruby $ rvm install 1.9.3 $ rvm use 1.9.3 $ gem install collins_shell
Setup
By default collins-shell looks for a configuration file in ~/.collins.yaml. You can
specify an alternate location to use with the --config=/path/to/collins.yaml option or using
the COLLINS=/path/to/collins.yaml environment variable. There is a sample config file below.
--- host: https://collins.example.org username: "joeuser" password: "secret" timeout: 30
If configuration files aren't your thing you can just all the variables from the config in via command options.
You can specify the host, username and password via the --host, --username,
and --password options. If you just specify --password (without a value), or your configs
are missing a password, you will be prompted for one. The password prompt masks your input.
Usage
Before using any command, check out the help. You can see help for a command by running:
collins-shell help <command>
collins-shell <sub> help <command>
<sub> is something like asset, ipmi, or ip_address and
<command> is something like create, delete or get.
> collins-shell
Tasks:
collins-shell asset <command> # Asset related commands
collins-shell console # drop into the interactive collins shell
collins-shell help [TASK] # Describe available tasks or one specific task
collins-shell ip_address <command> # IP address related commands
collins-shell ipmi <command> # IPMI related commands
collins-shell latest # check if there is a newer version of collins-shell
collins-shell log MESSAGE # log a message on an asset
collins-shell logs TAG # fetch logs for an asset specified by its tag. Use "all" for all logs
collins-shell power ACTION --reason=REASON --tag=TAG # perform power action (off, on, rebootSoft, rebootHard, etc) on an asset
collins-shell power_status # check power status on an asset
collins-shell provision <command> # Provisioning related commands
collins-shell state <command> # State management related commands - use with care
collins-shell tag <command> # Tag related commands
collins-shell version # current version of collins-shell
> collins-shell asset
Tasks:
collins-shell asset create --tag=TAG # create an asset in collins
collins-shell asset delete --reason=REASON --tag=TAG # delete an asset in collins (must be cancelled)
collins-shell asset delete_attribute KEY # delete an attribute in collins
collins-shell asset find --selector=key value # find assets using the specified selector
collins-shell asset find_similar TAG # get similar unallocated assets for a gven asset
collins-shell asset get TAG # get an asset and display its attributes
collins-shell asset help [COMMAND] # Describe subcommands or one specific subcommand
collins-shell asset set_attribute KEY VALUE # set an attribute in collins
collins-shell asset set_status --reason=REASON # set status, state, or both on an asset
> collins-shell asset help get
Usage:
collins-shell asset get TAG
Options:
[--config=CONFIG] # YAML configuration file
[--debug] # Debug output
[--host=HOST] # Collins host (e.g. http://host:port)
[--password=PASSWORD] # Collins password
[--quiet] # Be quiet when appropriate
[--timeout=N] # Collins client timeout
# Default: 30
[--username=USERNAME] # Collins username
[--confirm] # Require exec confirmation. Defaults to true
# Default: true
[--exec=EXEC] # Execute a command using the data from this asset. Use {{hostname}}, {{ipmi.password}}, etc for substitution
[--logs] # Also display asset logs
[--remote=REMOTE] # Remote location to search. This is a tag in collins corresponding to the datacenter asset
get an asset and display its attributes
-
Collins Get
The collins shell provides you with all the same information available to you in the web UI, in an easy to digest format.
Universal Options
The following list of options can be specified with every collins shell command. In some
cases (development) it may be useful to set the environment variable COLLINS_DEBUG=true
which will turn on HTTP level debugging. Use this in conjunction with the --debug option.
--config=CONFIG # YAML configuration file --debug # Debug output --host=HOST # Collins host (e.g. http://host:port) --quiet # Mostly used in conjunction with commands that have an --exec option --timeout=SECONDS # Seconds to allow for operation, defaults to 30 --username=USERNAME # Collins username --password=PASSWORD # Collins password
Common Options
Many options will allow you to specify either a selector (matching many assets) or a tag (matching a single asset). A tag is the asset tag, a selector is a space separated list of key:value pairs that are asset keys and values.
# Allocated web servers (note that multiple selectors are separated by a space) --selector=hostname:'^web.*' status:Allocated # Allocated master database servers in the main pool --selector=primary_role:database pool:main secondary_role:master status:Allocated # Asset with tag 001923 --tag=001923
Tips and Tricks
There are useful examples in the recipes guide.