Overview

The Collins data model

Collins models your infrastructure through the use of "assets". An asset can be anything you want, but the pre-configured asset types are Servers, Chassis, Switch, Router, and Configuration. By composing these objects with a rich tagging/attribute system, IPAM, status and state transitions, and logging system, it allows you to dynamically model and automate your infrastructure, no matter the size of your organization.

• Assets
• Tags (or Attributes)
• Status & States
• Logs
• Addresses

Assets

An asset describes a piece of your infrastructure

Assets are simple representations of a part of your infrastructure topology. They can be things like Routers and Switches, or Servers, Chassis, Configuration, Datacenter, or even user-defined types for your specific needs. Assets primarily consist of:

Asset Tag

A tag is a unique, immutable, alphanumeric sequence of characters associated with the asset. These are arbitrary but nearly all API calls for interacting with assets will reference this tag. For servers, these are generally burned into the FRU/DMI and identify the asset. For physical assets, it is very useful to mark the asset tag on the outside of the device.

Status

Indicates the current lifecycle phase of the asset. More details in Status & State

State

Indicates a phase (status) specific state of the asset. Operators can define arbitrary states with the API. More details in the next section.

Type

Describes the type of the asset. Of primary interest to most users will be server nodes and configurations.

Tags

Also known as attributes, these are user defined key value pairs that are associated with an asset. These can be used for grouping assets in queries, performing conditional automation actions, informing business logic during provisioning, etc.

New asset types can be created via the Asset Type API. The preconfigured types that Collins ships with are:

• Server Node
• Rack
• Switch
• Router
• Power Circuit
• Power Strip
• Data Center
• Configuration
• more via the Asset Type API

Tags

Multi-dimensional key/values for an asset

The most fundamentally important concept in collins is that of a tag. A tag is a key value pair, with an optional dimension index. For example, PORT tag has a default index of 0, but could be N-dimensional as you define multiple PORT tags with different dimensions. You are not limited by a predefined set of tags for your assets! If you tag an asset with a new tag FRESHNESS=9000, it will be immediately available to query. If you set a new tag value using the same dimension, the old value will be overwritten. All tag changes, including deletes, are logged as part of the asset audit trail.

Tag Types

Tags can be either managed, automated, or unmanaged.

Managed

Tags that are not user managable (such as CPU speed) and can only be set or changed during certain lifecycle events, such as the server intake process updating LSHW/LLDP information. These tags can not be changed, updated or deleted by a user through the normal tag API. This is enforced in Collins.

Automated

Tags that are managed by external automated processes such as server provisioning. Although these can technically be set/changed by users via the API, convention dictates that they should not be. It is up to you to set expectations, and manage automated tags in your automation software. For example, tags in deleteSomeMetaOnRepurpose (see Features Configuration) can be used to automatically wipe specific automated tags when an asset is reprovisioned.

Unmanaged

Tags set by a user via some api interaction (most likely), but probably on a manual basis. In some cases these may be part of an automated process, but some user-interaction will dictate the tag value. These are typically used for things like notes, links, testing, etc.

Dimensions

By default, a simple tag has the "0" dimension. A dimension can be any integer value (including a timestamp) and is typically used for grouping things together. You can imagine the disks of a host being described like

{
  "0": {
    "DEV_NAME": "/dev/sda",
    "TOTAL_DISK_SPACE_BYTES": 536870912000,
    "DISK_TYPE": "SATA"
  },
  "1": {
    "DEV_NAME": "/dev/sdb",
    "TOTAL_DISK_SPACE_BYTES": 0,
    "DISK_TYPE": "CD-ROM"
  }
}

This information tells you that the asset has two disks (one sata, one cd-rom) and tells you a bit about each physical disk. The value of the dimension in this case is just to logically group things together, and possibly to provide some ordering. You could also however use the dimension to represent things like port allocation (i.e. each dimension represents a port in use on the asset), or order of some arbitrary event where the timestamp is the dimension used. It's up to you.

Tag Conventions

The flexible and free-form nature of using user-defined tags to drive automation and add semantic value to Assets in Collins dictates that users agree upon some conventions for tag usage. For example, if you use the PUPPET_ENVIRONMENT tag to override which puppet environment an asset will use when running puppet, external automation should be put in place to ensure environments are cleaned up after use, and that assets do not remain in non-production environments for too long.

Status & state

Status and state describe the current phase of the lifecycle of an asset.

Status

The lifecycle (from birth to death) of an asset are described in terms of its status. The possible status values are fixed and can not be managed via the API. While all available status values are listed below, the descriptions given are primarily indiciative of their meanings for a server. A non-server asset type such as a configuration may only ever be allocate d or decommissioned, for instance. The status values described below also give some specific insight into the Tumblr intake process for hardware.

Incomplete

Host not yet ready for use. It has been powered on and entered in Collins but the automated induction process is still being performed.

New

Host has completed the automated induction process and is waiting for an onsite tech to complete physical intake

Unallocated

Host has completed intake process and is ready for use (eg available resource for provisioning into a role)

Provisioning

Host has started provisioning process but has not yet completed it

Provisioned

Host has finished provisioning and is awaiting final automated verification

Allocated

This asset is in what should likely be considered a production state

Cancelled

Asset is no longer needed and is awaiting decommissioning (typically only associated with SoftLayer and other transient cloud-provided assets)

Decommissioned

Asset has completed the outtake process and can no longer be managed

Maintenance

Asset is undergoing some kind of maintenance and should not be considered for production use

The status transition should not generally happen by hand. Automated processes should drive status changes, not people. In fact, the collins UI only allows you to change the status by taking an action (e.g. cancelling an asset or putting it into maintenance).

State

While the status of an asset describes where it is in a discrete lifecycle, the state describes a lifecycle specific to a status. For example, a server that is in maintenance may have a state of HARDWARE_PROBLEM or HARDWARE_UPGRADE. Also note that those states are not appropriate for healthy (non-maintenance) assets, and so these states are restricted to assets with a status of Maintenance. New states may be defined with this API.

A state can be either a system state, or a non-system state. System states can not be modified or destroyed. Non-system states can be modified and destroyed. Via the API you can only create non-system states, although support for adding system states may be added in the future. A state can be bound to a status (such as the case of HARDWARE_PROBLEM), or can be used with any status (such as the case of RUNNING). The out of the box available states are described below.

A few useful examples of user-defined states are as follows:

Provisioned:PENDING_ALLOCATION

Provisioned:PENDING_ALLOCATION could represent the state an asset enters when it has finished its automated provisioning (i.e. base OS imaging, configuration management application, and application deployments), but has not yet been validated to be healthy in external monitoring systems. For example, an "Allocator" script could collect all assets that are Provisioned:PENDING_ALLOCATION and perform validation checks on them (are they in DNS? are they pingable? SSH? are metrics being collected in Prometheus? Are all alerts in Prometheus green?). When health is verified, then the Allocator can move the asset into Allocated where it will be automatically pulled into service discovery/load balancers.

Allocated:DRAINING

Another useful state could be Allocated:DRAINING. This state could reflect an asset is currently taking production traffic and should continue to be monitored, but signals to loadbalancers and resource managers to stop routing new jobs/requests to this asset. This is especially useful for Hadoop Datanodes. Once the asset is fully drained of jobs, an automated process could move it from Allocated:DRAINING into a Maintenance state to pull it out of rotation fully. At Tumblr, we use these states to programmatically generate HDFS dfs.hosts and dfs.exclude lists to dynamically manage the participating members of the Hadoop cluster, and automatically Decommission any asset in Allocated:DRAINING.

Logs

An audit trail with an API

Every modification or lifecycle event that occurs with an asset is logged, along with who made the change and the time of the change. If a tag is modified (and not encrypted), the previous and new value are both stored. Logs can be searched via the API and can be viewed on the web as well. Logs are immutable but can be created via the API. Below is a list of log levels (based on syslog) and descriptions.

System logs (messages that aren't specific to any particular kind of asset) can only be logged internally by collins. Collins of course uses an asset to log these kinds of messages against. By default the system asset is the multicollins.thisInstance value, or tumblrtag1. You can specify the system asset via the features.syslogAsset configuration.

Addresses

IPAM for engineers, API included

Collins has an IP Address Management (IPAM) system built into it. The IPAM system is used for allocating both IPMI addresses and typical addresses. Addresses are configured in pools (which typically correspond to a VLAN), but can also be configured to be pool-less in the case where you don't manage your own IP Address space.

Collins will prevent duplicate IP address allocation, and will almost always use the smallest available address in a range. It is possible to allocate an address against any kind of asset. This is sometimes useful for instance when managing a VIP (virtual or floating IP address). You can create a configuration asset that holds the VIP for a service, then link that asset to others that will actually share the address.

In addition to address allocation, collins provides the ability to do other things you would expect from a typical IPAM system such as querying used addresses, understanding what an IP space looks like, finding assets in a pool or by address, etc.

At Tumblr we combine the IPAM functionality of Collins with the per asset LLDP data to automatically manage switch provisioning. We also use this data for generating kickstart files with the correct address information.

The fundamental idea with Collins IPAM is that of a pool. A pool is a named group of addresses. A pool definition will specify the network address range (specified in CIDR notation), an optional start address (e.g. the IP to start allocating from in the specified range), an optional gateway (if it's not the one you would infer from the CIDR range), and a name. Once a pool is configured it is possible to allocate addresses in that pool. If you don't manage your own address space, no worries. You can operate in a a 'pool-less' mode where you can specify any address.

There is more information available in the API section as well in the configuration section to configure address pools.