Collins Concepts

Stuff to know before getting started

Overview

The Collins data model

Collins models your infrastructure through the use of "assets". An asset can be anything you want, but the pre-configured asset types are Servers, Chassis, Switch, Router, and Configuration. By composing these objects with a rich tagging/attribute system, IPAM, status and state transitions, and logging system, it allows you to dynamically model and automate your infrastructure, no matter the size of your organization.

Assets

An asset describes a piece of your infrastructure

Assets are simple representations of a part of your infrastructure topology. They can be things like Routers and Switches, or Servers, Chassis, Configuration, Datacenter, or even user-defined types for your specific needs. Assets primarily consist of:

Asset Tag
A tag is a unique, immutable, alphanumeric sequence of characters associated with the asset. These are arbitrary but nearly all API calls for interacting with assets will reference this tag. For servers, these are generally burned into the FRU/DMI and identify the asset. For physical assets, it is very useful to mark the asset tag on the outside of the device.
Status
Indicates the current lifecycle phase of the asset. More details in Status & State
State
Indicates a phase (status) specific state of the asset. Operators can define arbitrary states with the API. More details in the next section.
Type
Describes the type of the asset. Of primary interest to most users will be server nodes and configurations.
Tags
Also known as attributes, these are user defined key value pairs that are associated with an asset. These can be used for grouping assets in queries, performing conditional automation actions, informing business logic during provisioning, etc.

New asset types can be created via the Asset Type API. The preconfigured types that Collins ships with are:

  • Server Node
  • Rack
  • Switch
  • Router
  • Power Circuit
  • Power Strip
  • Data Center
  • Configuration
  • more via the Asset Type API

Tags

Multi-dimensional key/values for an asset

The most fundamentally important concept in collins is that of a tag. A tag is a key value pair, with an optional dimension index. For example, PORT tag has a default index of 0, but could be N-dimensional as you define multiple PORT tags with different dimensions. You are not limited by a predefined set of tags for your assets! If you tag an asset with a new tag FRESHNESS=9000, it will be immediately available to query. If you set a new tag value using the same dimension, the old value will be overwritten. All tag changes, including deletes, are logged as part of the asset audit trail.

Tag Types

Tags can be either managed, automated, or unmanaged.

Managed
Tags that are not user managable (such as CPU speed) and can only be set or changed during certain lifecycle events, such as the server intake process updating LSHW/LLDP information. These tags can not be changed, updated or deleted by a user through the normal tag API. This is enforced in Collins.
Automated
Tags that are managed by external automated processes such as server provisioning. Although these can technically be set/changed by users via the API, convention dictates that they should not be. It is up to you to set expectations, and manage automated tags in your automation software. For example, tags in deleteSomeMetaOnRepurpose (see Features Configuration) can be used to automatically wipe specific automated tags when an asset is reprovisioned.
Unmanaged
Tags set by a user via some api interaction (most likely), but probably on a manual basis. In some cases these may be part of an automated process, but some user-interaction will dictate the tag value. These are typically used for things like notes, links, testing, etc.

Dimensions

By default, a simple tag has the "0" dimension. A dimension can be any integer value (including a timestamp) and is typically used for grouping things together. You can imagine the disks of a host being described like

{
  "0": {
    "DEV_NAME": "/dev/sda",
    "TOTAL_DISK_SPACE_BYTES": 536870912000,
    "DISK_TYPE": "SATA"
  },
  "1": {
    "DEV_NAME": "/dev/sdb",
    "TOTAL_DISK_SPACE_BYTES": 0,
    "DISK_TYPE": "CD-ROM"
  }
}

This information tells you that the asset has two disks (one sata, one cd-rom) and tells you a bit about each physical disk. The value of the dimension in this case is just to logically group things together, and possibly to provide some ordering. You could also however use the dimension to represent things like port allocation (i.e. each dimension represents a port in use on the asset), or order of some arbitrary event where the timestamp is the dimension used. It's up to you.

Tag Conventions

The flexible and free-form nature of using user-defined tags to drive automation and add semantic value to Assets in Collins dictates that users agree upon some conventions for tag usage. For example, if you use the PUPPET_ENVIRONMENT tag to override which puppet environment an asset will use when running puppet, external automation should be put in place to ensure environments are cleaned up after use, and that assets do not remain in non-production environments for too long.

Manual Tags

These tags are key value pairs used by humans to coordinate activity. Generally, they reflect some static-ish attributes about a system (for example, HAS_GPU=true could be used to signal to systems that an asset contains at least one GPU. More commonly, humans set these tags on assets in a transitive manner, to mark and later query for a set of assets to work on. For example, a user may mark 20 assets with CLAIMED=gabe and later query for assets with CLAIMED=gabe to reprovision these hosts into new roles.

Automated Tags

These are unmanaged tags, that are maintained/updated by automated processes.

General Purpose Automated Tags

Non-tumblr specific automated tags, although some of the tools may be Tumblr specific.

TagPopulated ByDescription
HOSTNAME your provisioning system Populated when an asset successfully starts the provisioning process. Automation can generate a new hostname based on roles/pools, etc.
NODECLASS your provisioning system Populated when an asset successfully starts the provisioning process. This can be used by the puppet master to resolve the nodeclass for a host in the external node classifier.
CONTACT collins at provision time When a user provisions an asset the specified CONTACT is populated. This is used for notification of maintenance, provisioning being done, etc.
PRIMARY_ROLE collins When a user provisions an asset the chosen PRIMARY_ROLE is populated. This is used to communicate the intended purpose of the asset (CACHE, DATABASE, HADOOP, PROXY, WEB, etc)
POOL collins Populated along with PRIMARY_ROLE, this tag is used to communicate the functional pool the asset should participate in, e.g. MEMCACHE_POST_POOL
SECONDARY_ROLE collins Populated along with PRIMARY_ROLE, this tag is used to communicate some secondary role such as MASTER, SLAVE, or a deployment group (such as WEB_A, WEB_B).
SYSTEM_USERNAME your provisioning system You can specify the administrative username here.
SYSTEM_PASSWORD your provisioning system Randomly generate new root passwords at provision time. You can use something like puppet to enforce the password as well.
MAINTENANCE_TICKET collins If the asset is in maintenance mode, this can show the ticket created to track the hardware's repair

Most hardware information is gleaned from LSHW and most networking information is taken from LLDP. The usefulness of the hardware information is dependent on the version of LSHW (recent versions get better information).

Managed tags automatically use an additional dimension if required. For example, say you have two memory sticks and we're tracking a description and size for each stick. Collins will create two 'groups', one that represents each memory stick. This is how collins differentiates between the same tag being used multiple times for the same asset.

General Information

All of these are specified as HTTP parameters during the intake process. These are not inferred, but specified.

TagTypeDescription
SERVICE_TAG String Vendor service tag (e.g. dell tag)
CHASSIS_TAG String Manufacturer serial number
RACK_POSITION String Physical position in the rack
POWER_PORT_A String The port in the CDU that the asset is attached to. The name and value will depend on your power configuration (detailed in configuration)

CPU Information

Collins assumes that each CPU is identical so only uses one dimension to represent all CPU's. Collins creates the visual hardware representation based on the CPU_COUNT. This information is taken from LSHW.

TagTypeDescription
CPU_COUNT Integer Physical CPU count in machine
CPU_CORES Integer Number of cores per physical CPU
CPU_THREADS Integer Number of threads per core
CPU_SPEED_GHZ Float Speed of CPU
CPU_DESCRIPTION String Vendor description or label

Memory Information

Collins represents a stick of RAM using the dimension (dictates bank used), MEMORY_SIZE_BYTES and MEMORY_DESCRIPTION. Each dimension represents a stick of RAM present in an asset. Collins creates the visual representation of the RAM layout by using the MEMORY_BANKS_TOTAL and filling in the sticks that are missing. No additional data is stored on missing RAM. This information is taken from LSHW.

TagTypeDescription
MEMORY_SIZE_BYTES Long Size of single RAM module
MEMORY_DESCRIPTION String Vendor description or label
MEMORY_SIZE_TOTAL Long Total number of bytes (MEMORY_SIZE_BYTES * number of modules)
MEMORY_BANKS_TOTAL Integer Total number of RAM banks

Network Information

Collins represents a NIC with its NIC_SPEED, MAC_ADDRESS and NIC_DESCRIPTION. Each dimension represents a different physical network card. This information is taken from LSHW.

TagTypeDescription
NIC_SPEED Long Speed in bits per second
MAC_ADDRESS String MAC address of network interface
NIC_DESCRIPTION String Vendor description or label

Disk Information

Collins represents a disk with its DISK_SIZE_BYTES, DISK_TYPE and DISK_DESCRIPTION. Each dimension represents a different physical disk. This information is taken from LSHW.

TagTypeDescription
DISK_SIZE_BYTES Long Size in bytes of disk. This will occasionally report as 0 when the disk is part of a RAID group, or in the case of a CD-ROM. The number of physical disks and storage total will still be accurate in this case.
DISK_TYPE Enum Type of disk. IDE, SCSI, PCIe, or CD-ROM.
DISK_DESCRIPTION String Vendor description or label
DISK_STORAGE_TOTAL String Sum of size for all disks. Because this number of often larger than a Long, it is stored as a String

LLDP Information

Collins represents each interface connected to an LLDP capable device using all of the information below. Each dimension represents a different physically connected network interface. Each interface may have zero or more VLAN's. This information is taken from LLDP.

  • LLDP_INTERFACE_NAME
  • LLDP_CHASSIS_NAME
  • LLDP_CHASSIS_ID_TYPE
  • LLDP_CHASSIS_ID_VALUE
  • LLDP_CHASSIS_DESCRIPTION
  • LLDP_PORT_ID_TYPE
  • LLDP_PORT_ID_VALUE
  • LLDP_PORT_DESCRIPTION
  • LLDP_VLAN_ID
  • LLDP_VLAN_NAME

Status & state

Status and state describe the current phase of the lifecycle of an asset.

Status

The lifecycle (from birth to death) of an asset are described in terms of its status. The possible status values are fixed and can not be managed via the API. While all available status values are listed below, the descriptions given are primarily indiciative of their meanings for a server. A non-server asset type such as a configuration may only ever be allocate d or decommissioned, for instance. The status values described below also give some specific insight into the Tumblr intake process for hardware.

Incomplete

Host not yet ready for use. It has been powered on and entered in Collins but the automated induction process is still being performed.

New

Host has completed the automated induction process and is waiting for an onsite tech to complete physical intake

Unallocated

Host has completed intake process and is ready for use (eg available resource for provisioning into a role)

Provisioning

Host has started provisioning process but has not yet completed it

Provisioned

Host has finished provisioning and is awaiting final automated verification

Allocated

This asset is in what should likely be considered a production state

Cancelled

Asset is no longer needed and is awaiting decommissioning (typically only associated with SoftLayer and other transient cloud-provided assets)

Decommissioned

Asset has completed the outtake process and can no longer be managed

Maintenance

Asset is undergoing some kind of maintenance and should not be considered for production use

The status transition should not generally happen by hand. Automated processes should drive status changes, not people. In fact, the collins UI only allows you to change the status by taking an action (e.g. cancelling an asset or putting it into maintenance).

State

While the status of an asset describes where it is in a discrete lifecycle, the state describes a lifecycle specific to a status. For example, a server that is in maintenance may have a state of HARDWARE_PROBLEM or HARDWARE_UPGRADE. Also note that those states are not appropriate for healthy (non-maintenance) assets, and so these states are restricted to assets with a status of Maintenance. New states may be defined with this API.

A state can be either a system state, or a non-system state. System states can not be modified or destroyed. Non-system states can be modified and destroyed. Via the API you can only create non-system states, although support for adding system states may be added in the future. A state can be bound to a status (such as the case of HARDWARE_PROBLEM), or can be used with any status (such as the case of RUNNING). The out of the box available states are described below.

StatusState LabelState NameState Description
Any Failed FAILED A service in this state has encountered a problem and may not be operational. It cannot be started nor stopped.
Any New NEW A service in this state is inactive. It does minimal work and consumes minimal resources.
Any Running RUNNING A service in this state is operational.
Any Starting STARTING A service in this state is transitioning to Running.
Any Stopping STOPPING A service in this state is transitioning to Terminated.
Any Terminated TERMINATED A service in this state has completed execution normally. It does minimal work and consumes minimal resources.
Maintenance Hardware Problem HARDWARE_PROBLEM An asset is experiencing a non-IPMI issue and needs to be examined. It needs investigation.
Maintenance Hardware Testing HW_TESTING Performing some testing that requires putting the asset into a maintenance state.
Maintenance Hardware Upgrade HARDWARE_UPGRADE An asset is in need or in process of having hardware upgraded.
Maintenance IPMI Problem IPMI_PROBLEM An asset is experiencing IPMI issues and needs to be examined. It needs investigation.
Maintenance Maintenance NOOP MAINT_NOOP Doing nothing, bouncing this through maintenance for my own selfish reasons.
Maintenance Network Problem NETWORK_PROBLEM An asset is experiencing a network problem that may or may not be hardware related. It needs investigation.
Maintenance Relocation RELOCATION An asset is being physically relocated.

A few useful examples of user-defined states are as follows:

Provisioned:PENDING_ALLOCATION

Provisioned:PENDING_ALLOCATION could represent the state an asset enters when it has finished its automated provisioning (i.e. base OS imaging, configuration management application, and application deployments), but has not yet been validated to be healthy in external monitoring systems. For example, an "Allocator" script could collect all assets that are Provisioned:PENDING_ALLOCATION and perform validation checks on them (are they in DNS? are they pingable? SSH? are metrics being collected in Prometheus? Are all alerts in Prometheus green?). When health is verified, then the Allocator can move the asset into Allocated where it will be automatically pulled into service discovery/load balancers.

Allocated:DRAINING

Another useful state could be Allocated:DRAINING. This state could reflect an asset is currently taking production traffic and should continue to be monitored, but signals to loadbalancers and resource managers to stop routing new jobs/requests to this asset. This is especially useful for Hadoop Datanodes. Once the asset is fully drained of jobs, an automated process could move it from Allocated:DRAINING into a Maintenance state to pull it out of rotation fully. At Tumblr, we use these states to programmatically generate HDFS dfs.hosts and dfs.exclude lists to dynamically manage the participating members of the Hadoop cluster, and automatically Decommission any asset in Allocated:DRAINING.

Logs

An audit trail with an API

Every modification or lifecycle event that occurs with an asset is logged, along with who made the change and the time of the change. If a tag is modified (and not encrypted), the previous and new value are both stored. Logs can be searched via the API and can be viewed on the web as well. Logs are immutable but can be created via the API. Below is a list of log levels (based on syslog) and descriptions.

LevelDescription
EMERGENCY A "panic" condition - notify all tech staff on call? (earthquake? tornado?) - affects multiple apps/servers/sites...
ALERT Should be corrected immediately - notify staff who can fix the problem - example is loss of backup ISP connection
CRITICAL Should be corrected immediately, but indicates failure in a primary system - fix CRITICAL problems before ALERT - example is loss of primary ISP connection
ERROR Non-urgent failures - these should be relayed to developers or admins; each item must be resolved within a given time
WARNING Warning messages - not an error, but indication that an error will occur if action is not taken, e.g. file system 85% full - each item must be resolved within a given time
NOTICE Events that are unusual but not error conditions - might be summarized in an email to developers or admins to spot potential problems - no immediate action required
INFORMATIONAL Normal operational messages - may be harvested for reporting, measuring throughput, etc - no action required
DEBUG Info useful to developers for debugging the application, not useful during operations
NOTE Creates by users via the web UI, can be any kind of message

System logs (messages that aren't specific to any particular kind of asset) can only be logged internally by collins. Collins of course uses an asset to log these kinds of messages against. By default the system asset is the multicollins.thisInstance value, or tumblrtag1. You can specify the system asset via the features.syslogAsset configuration.

Addresses

IPAM for engineers, API included

Collins has an IP Address Management (IPAM) system built into it. The IPAM system is used for allocating both IPMI addresses and typical addresses. Addresses are configured in pools (which typically correspond to a VLAN), but can also be configured to be pool-less in the case where you don't manage your own IP Address space.

Collins will prevent duplicate IP address allocation, and will almost always use the smallest available address in a range. It is possible to allocate an address against any kind of asset. This is sometimes useful for instance when managing a VIP (virtual or floating IP address). You can create a configuration asset that holds the VIP for a service, then link that asset to others that will actually share the address.

In addition to address allocation, collins provides the ability to do other things you would expect from a typical IPAM system such as querying used addresses, understanding what an IP space looks like, finding assets in a pool or by address, etc.

At Tumblr we combine the IPAM functionality of Collins with the per asset LLDP data to automatically manage switch provisioning. We also use this data for generating kickstart files with the correct address information.

The fundamental idea with Collins IPAM is that of a pool. A pool is a named group of addresses. A pool definition will specify the network address range (specified in CIDR notation), an optional start address (e.g. the IP to start allocating from in the specified range), an optional gateway (if it's not the one you would infer from the CIDR range), and a name. Once a pool is configured it is possible to allocate addresses in that pool. If you don't manage your own address space, no worries. You can operate in a a 'pool-less' mode where you can specify any address.

There is more information available in the API section as well in the configuration section to configure address pools.