Overview

Things to know before you configure

Collins configuration files are type safe. That is, you can't create a configuration that will lead to a runtime error. If collins detects a configuration error it will fail to start. If you modify a config after the system starts and that config is invalid, it will not load and collins will complain loudly. If this is not the case, please file a bug.

Every type of configuration comes with a reference config. Reference configuration files can be found in the github repo. Reference files are not modified by users, they serve to provide Collins with default values as well as to tell it what the appropriate value types are (lists, strings, etc). Most reference configurations come with some level of documentation as well. In general you should use and make modifications to conf/application.conf.

The configuration file that you create for collins is overriding the values found in the reference configs. There is a sample configuration in the github repo for you to use as a starting point.

You can pass your configuration file to collins by using a system property. Pass -Dconfig.file=/path/to/file will cause that file to be used for the application.

General

Configuration options that don't fit anywhere else

Application Specific

These configuration aspects are specific to the play framework.

application.secret="AbFgHx0eJx8lalkja812389uasdlkajsdlka98012398uasdlkasdklajsd81298"
parsers.text.maxLength=1048576
evolutionplugin = disabled

Crypto

The only crypto configuration available is the key. The crypto key must not be changed after the system is initialized for the first time or you will be unable to decrypt data.

crypto.key = "lkkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdlkajsdla"

Database

db {
  collins {
    logStatements = false
    user = "sa"
    password = ""
    driver = "org.h2.Driver"
    url = "jdbc:h2:mem:play;IGNOERCASE=true"
  }
}

db {
  collins {
    logStatements = false
    user = "root"
    password = ""
    driver = "com.mysql.jdbc.Driver"
    url = "jdbc:mysql://localhost/collins?autoReconnect=true&interactiveClient=true"
  }
}

LSHW Parsing

Due to how LSHW handles PCIe flash devices, you have to give the parser a hint as to which attribute values indicate that the device is in fact a flash storage device. Here you can specify the string to match on and the size to use if the match is found.

lshw {
  defaultNicCapacity = 10000000000
  flashProducts = ["flashmax","fusionIO","TachION Drive"]
  flashSize = 1400000000000
}

Query Log

The query log is primary used for development but allows you to turn on database query logging and API (find/search) logging.

querylog {
  enabled = false
  includeResults = false
  prefix = "QUERY: "
  frontendLogging = false
}

Authentication

Setting up user authentication and authorization

Authentication Namespace authentication

Collins supports a variety of authentication backends along with a flexible API endpoint based permissions system. Some features also have specific permissions.

Collins comes bundled with a small in-memory authentication mechanism for testing, file based authentication for small or simple installations, and LDAP based authentication for large or enterprise installations. If using LDAP we recommend running an LDAP slave on the collins node itself in order to be resilient to large outages where the usual LDAP infrastructure might be unavailable.

The permissions example below creates two user groups, admins and tools. User groups are just named groups of users that are stored in the YAML file. This is useful for people authenticating with flat files or if you want to store some group information here. The admins user group has two users, alan and bmatheny. The tools user group has three users. There are two feature permissions specified, one for rate limiting and the other for who can see encrypted tags, like passwords. The last permission is for the powerStatus API endpoint and is open to a broader group of users. Every API permission name is documented in the API documentation.

users:
  admins:
    - "alan"
    - "bmatheny"
  tools:
    - "jetpants"
    - "itouch"
    - "phil"
permissions:
  feature.noRateLimit:
    - "u=tools"
  feature.canSeeEncryptedTags:
    - "u=admins"
    - "u=tools"
    - "g=infra"
  controllers.AssetManagementApi.powerStatus:
    - "g=infra"
    - "g=ops"
    - "u=admins"
    - "u=tools"

authentication {
  type = default
  permissionsFile = "conf/permissions.yaml"
  permissionsCacheSpecification = "expireAfterWrite=30s"
  adminGroup = [Infra]
}

LDAP Authentication Namespace authentication.ldap

At Tumblr LDAP is the primary authentication mechanism. We have tested that this code works with at least RFC 2307, IPA, and the OS X admin server.

authentication {
  type = ldap
  ldap {
    host = "localhost"
    groupAttribute = "uniqueMember"
    searchbase = "dc=example,dc=org"
    usersub = "ou=people"
    groupsub = "ou=groups"
    cacheSpecification = "expireAfterWrite=30s"
  }
}

File Authentication Namespace authentication.file

The file format used is a variation on that used by Apache. You can find a sample script for generating users and passwords in scripts/gen_passwords.sh.

authentication {
  type = file
  file {
    userfile = "conf/users.conf"
    cacheSpecification = "expireAfterWrite=30s"
  }
}

Mixed Authentication

Mixed authentication allows collins to use multiple authentication backends simultaneously. An example use-case could be a scenario where real users live in LDAP but system users (for automation etc.) are defined using file authentication.

To use mixed authentication, simply specify a list of authentication backends as the type, and add configuration (as described above) for each of them within the authentication block. A simple example can be seen below.

The mixed authentication provider will try each of the methods in the order listed and use the first one that returns a successful match.

authentication {
  type = "file,ldap"

  file {
    userfile = "conf/users.conf"
  }

  ldap {
    host = "localhost"
    groupAttribute = "uniqueMember"
    searchbase = "dc=example,dc=org"
    usersub = "ou=people"
    groupsub = "ou=groups"
  }
}

Cache

Caching used by collins

Collins uses caching in several places, most notably to cache ldap responses, metrics, and to coordinate across multiple nodes for the firehose.

cache {
  enabled=true

  type=in-memory
}

Guava in-memory cache

guava {
  enabled=true

  # used when deploying in standalone mode
  specification="maximumSize=10000,expireAfterWrite=10s,recordStats"
}

Hazelcast distributed cache

hazelcast {
      enabled=false

      # http://docs.hazelcast.org/docs/3.5/manual/html/tcp.html
      # used when deploying in clustered mode
      configFile="conf/hazelcast/hazelcast.xml"
      members=""
    }
}

Callbacks

Hooking into lifecycle events in collins

Namespace callbacks

Collins supports a callback mechanism that allows you to trigger events when the configured callbacks are matched. A short example is probably most useful here.

callbacks.registry {
  hardwareProblem {
    on = "asset_update"
    when {
      previous.state = "!isMaintenance"
      current.state = "isMaintenance"
      current.states = [ HARDWARE_PROBLEM ]
    }
    action {
      type = exec
      command = "/sbin/notify <tag> sysadmin@example.org"
    }
  }
}

In the above configuration we have registered a callback named hardwareProblem. This callback will be triggered when an asset is moved to have a status of Maintenance with a state of HARDWARE_PROBLEM. When the callback is triggered we will execute a notify command that will email a systems administrator.

A callback is triggered by an event which is specified via the on config value. The value is then matched using a matcher which is specified via the when config value. If the matcher succeeds the specified action is taken.

callbacks {
  enabled = true
  class = "collins.callbacks.CallbackManagerPlugin"
  registry {
  }
}

Registry Namespace callbacks.registry

The registry holds all your configured callbacks.

Features

Collins feature flags

Namespace features

There are a number of parts of collins that you may wish to customize based on your environment. These are referred to in the reference configuration and elsewhere as feature flags. Available feature flags are documented below.

features {
  allowTagUpdates = []

  allowedServerUpdateStatuses = []

  defaultLogType = Informational

  deleteIpmiOnDecommission = true

  deleteIpAddressOnDecommission = true

  deleteMetaOnDecommission = false

  deleteSomeMetaOnRepurpose = []

  encryptedTags = []

  ignoreDangerousCommands = []

  intakeSupported = true

  hideMeta = []

  noLogAssets = []

  noLogPurges = []

  sloppyStatus = true

  sloppyTags = false
}

features {
  allowedServerUpdateStatuses = [ PROVISIONED, PROVISIONING ]

  encryptedTags = [SYSTEM_PASSWORD, LOCATION]

  noLogAssets = [chatty_config, other_chatty_config]

  noLogPurges = [SYSTEM_PASSWORD]
}

Views

Generic framing support for external tools

Overview Namespace views

In order to generically support external utilities to provide supplemental data for assets, a generic view system was implemented to allow operators to frame in arbitrary sites. This supplants the graphs plugin, and is used to provide easy access to tooling like Grafana, Nagios/Icinga, Prometheus, etc.

Frames Namespace views.frames

views {
  enabled = true
  frames = {
		# frame in grafana
    graph {
      enabled = true
      title = "Grafana"
      style = "width: 100%;height: 1200px;"
      script = """
        // if the asset is a server, and has a hostname, enable this tab
        function isEnabled(inp) {
          var obj = JSON.parse(inp);
          return ((obj.ASSET.TYPE === "SERVER_NODE" || obj.ASSET.TYPE === "SWITCH") && ("HOSTNAME" in obj.ATTRIBS[0]));
        }

        // all assets use the host-stats dashboard
        function getUrl(inp) {
          var obj = JSON.parse(inp);
          var dash = "host-stats"
          if (obj.ASSET.TYPE === "SWITCH") {
            dash = "switch-stats";
          }
          return "https://grafana.company.net/dashboard/db/" + dash + "?var-hostname=" + obj.ATTRIBS[0]["HOSTNAME"];
        }
      """
    }

    # frame in nagios
    monitoring {
      enabled = true
      title = "Nagios"
      style = "width: 100%;height: 1200px;"
      script = """
        function isEnabled(inp) {
          var obj = JSON.parse(inp);
          return ((obj.ASSET.STATUS === "SERVER_NODE") && ("HOSTNAME" in obj.ATTRIBS[0]));
        }
        function getUrl(inp) {
          var obj = JSON.parse(inp);
          return "https://nagios.company.net/nagiosxi/includes/components/xicore/status.php?show=hostdetail&host=" + obj.ATTRIBS[0]["HOSTNAME"];
        }
      """
    }
  }
}

The above example configures 2 tabs. One frames in Grafana, showing a different dashboard depending on if the asset is a Server or Switch. The other frames in Nagios for the Server asset. This can be trivially extended to support PagerDuty, Prometheus, Alertmanager, Sensu, and any other monitoring, graphing, or alerting system. All that is required is the ability to parameterize the URL to the system with some identifying attribute of the host (hostname, IP, asset tag, etc).

Graphs

Display trending data

Overview Namespace graph

Collins has an easy to implement plugin for graph and trending systems, which can be extended to support any framed in HTML content. The graph plugin will display embedded graphs or charts for assets, or just links to those systems (depending on the graph plugin). Once enabled, you will find an additional tab on the asset details screen named 'Graphs' with all the configured information available in it.

There is currently support for Fibr, the OpenTSDB visualization tool we built at Tumblr (currently closed source!), and ganglia. There are no plans to opensource Fibr at this time.

Fibr Graphs Namespace graph.FibrGraphs

At Tumblr, Fibr is our primary trending, graphing, and systems visualization tool. NOTE: this has been deprecated in favor of the more generalized framing system.

graph.FibrGraphs {
  customMetrics = {
    mysqlHosts {
      selector = "PRIMARY_ROLE = \"DATABASE\""
      metrics = [
        MYSQL/COMMANDS,
        MYSQL/SLOW,
        MYSQL/THREADS,
        MYSQL/NET
      ]
    }
    memcacheHosts {
      selector = "POOL = MEMCACHE*"
      metrics = [
        MEMCACHE/COMMANDS,
        MEMCACHE/EVICTS,
        MEMCACHE/ITEMS,
        MEMCACHE/HITRATIO
      ]
    }
  }
}

In the above example, hosts that have a PRIMARY_ROLE of DATABASE will display in addition to the default metrics, the MySQL specific metrics. Assets with a POOL matching MEMCACHE* will additionally display memcache specific metrics.

Ganglia Graphs Namespace graph.GangliaGraphs

Ganglia is a distributed metrics generation, collection, and visualization system. Collins integration relies on recent ad-hoc-views support.

graph {
  class = "collins.graphs.GangliaGraphs"
  enabled = true
  GangliaGraphs {
    url = "http://my-ganglia.domain.local/"
 }
}

Ip address

IPAM Configuration

Namespace ipAddresses

The Collins IPAM configuration is pretty straight forward. You specify more or less a set of pools that are available for allocation. Once configured, assets can have addresses from any of these pools assigned to them.

ipAddresses {
  pools {
    DEV {
      startAddress = "172.16.4.4"
      network = "172.16.4.0/28"
    }
    ROW_1_1 {
      name = "ROW_1.1"
      startAddress = "172.16.64.5"
      network = "172.16.64.0/24"
    }
  }
}

The above configuration now allows you to allocate addresses out of two pools, a DEV pool and a ROW_1.1 pool. Notice that the ROW_1.1 pool had to specify a name. This is because an object name can not contain a period so must actually set the name manually as a string. The DEV pool did not have to do this. At Tumblr we have found pool names most useful for mapping to a particular VLAN and handling VLAN assignments using this information.

Pool Config

Each pool can take a name, gateway, startAddress and network.

Gpus

Collecting GPU information

Namespace gpu

GPU information can be stored in collins by adding GPU vendors to a whitelist. By default, any NVIDIA GPUs (PCI vendor = "NVIDIA Corporation") will be parsed from lshw XML data and added to collins. Additional vendors can be added by updating the configuration with their PCI vendor information.

gpu {

  supportedVendorStrings = ["NVIDIA Corporation", "Advanced Micro Devices, Inc. [AMD/ATI]"]

}

Ipmi

Configuring your IPMI network and authentication setup

Namespace ipmi

The IPMI configuration is used to drive generation of IPMI addresses, usernames and passwords. This information is used for power management operations such as power cycling an asset or lighting up an asset during the intake process.

ipmi {

  passwordLength = 12
  randomUsername = false
  username = "root"
  network = "10.0.0.0/16"
  startAddress = "10.0.0.3"

}

Lldp

Configuration for importing LLDP data

Namespace lldp

Collins accepts XML output from lldpctl in order to store data about which switch an asset is connected to.

lldp {
     requireVlanName = true
     requireVlanId   = false
}

Monitoring

Monitoring Collins

Namespace metricsreporter

Monitoring Collins using Coda Hale's Metrics and Chris Burrough's metrics-reporter-config project is relatively straightforward. See Metrics' documentation for how to report specific metrics, and the metrics-reporter-config documentation for how the configuration file should be formatted.

metricsreporter {

  enabled= false
  configFile = "conf/metrics-reporter-config.yaml"

}

Multicollins

For when a single data center just isn't enough

Namespace multicollins

When Tumblr grew into a second datacenter, we didn't want to bother with having to setup multi-master replication across sites or do some kind of master site replication scheme. Instead what we decided is that we would make each Collins instance aware of each other, and allow them to query and interact with each other. Don't worry, no crazy zookeeper magic going on here.

If you have multiple independent collins installations setup, to make them aware of each other you have to do two things.

1. Configure multi-collins
2. Create an appropriate asset in each collins instance

multicollins {
  enabled = false
  instanceAssetType = DATA_CENTER
  locationAttribute = LOCATION
  thisInstance = NONE
  queryCacheTimeout = 30
  cacheEnabled = true
}

Once you have configured multicollins you will now be able to use the remoteLookup API parameter for asset finds. The Ruby client automatically handles fetching remote assets when needed. Additionally, you should find a 'Remote Search' checkbox in the search UI.

Node classifier

Find similar assets

Namespace nodeclassifier

A pretty common thing to want to be able to do is ask collins to tell you what other assets are like another. At Tumblr we use this for bulk provisioning, where I might want to be able to do something like say 'provisioning me 100 servers that are just like this one'. This is the problem that is solved with the node classifier. A node classifier describes a node class (e.g. web, queue, database, etc) and allows you to query for assets described by the classifier.

Much like with the MultiCollins configuration, we reuse assets to configure our node classifiers. In order to create a node classifier create a new asset (we prefer Configuration assets), and set tags on it that describe the traits of similar assets. For instance you might create a Configuration asset named web_classifier and set the NIC_SPEED to 10000000000, the MEMORY_SIZE_TOTAL to 34359738368, and the TOTAL_DISK_STORAGE to 500107862016. This configuration now describes a web class host with a 1Gb NIC, 32GB of RAM, and a 500GB disk. Be sure to set an IS_NODECLASS tag to true on the asset.

With the above configuration specified you can now use the find similar API endpoint. You also in the web UI should find links for finding similar assets.

nodeclassifier {
  assetType = CONFIGURATION
  identifyingMetaTag = IS_NODECLASS
  displayNameAttribute = NAME
  excludeMetaTags = [IS_NODECLASS, NOTE]
  sortKeys = []
}

Power configuration

Power requirements for assets

Namespace powerconfiguration

Overview

Depending on the environment, each asset may have a different power configuration. A power configuration specifies the number of power units required per asset, the number of power components required per power unit, and various requirements for each power component.

Definitions

• An asset may have an associated power configuration
• A power configuration consists of one or more power units
• A power unit consits of one or more power components (e.g. power strip, power outlet, etc)
• A set of power units represents a complete power configuration

You can think of an asset power configuration as looking something like the following.

• PowerConfiguration
  • Power Unit 1
    • Power Component 1
    • Power Component 2
    • Power Component n
  • Power Unit 2
  • Power Unit n

Configuration

Note that this configuration heavily influences the intake process as well as what API parameters are available for asset updates. You can get a sample for what the asset looks like in the Asset Get JSON response.

Messages

It is possible to customize how the various components are labeled. The default bundled messages defines the following messages.

• powerconfiguration.unit.port.label - Power Strip {0}
• powerconfiguration.unit.outlet.label - Outlet {0}

Notice that the configured components each get a unit label. This allows you to customize how the intake screen looks.

Power managment

With great power, comes great responsibility

Namespace powermanagement

Assuming you would like the ability to perform power management operations on devices, such as power cycling them or powering them off, you will want to provide a power management configuration.

powermanagement {
  enabled = true
  command_template = "ipmitool -H <host> -U <username> -P <password> -I lan -L OPERATOR"

  allowAssetTypes = [SERVER_NODE]
  allowAssetsWithoutIpmi = false
  disallowStatus = []
  disallowWhenAllocated = [powerOff]

  commands {
    powerOff = ${powermanagement.command_template}" chassis power off"
    powerOn = ${powermanagement.command_template}" chassis power on"
    powerSoft = ${powermanagement.command_template}" chassis power soft"
    powerState = ${powermanagement.command_template}" chassis power status"
    rebootHard = ${powermanagement.command_template}" chassis power cycle"
    rebootSoft = ${powermanagement.command_template}" chassis power reset"
    identify = ${powermanagement.command_template}" chassis identify <interval>"
    verify = "ping -c 3 <host>"
  }
}

Note A command must be specified for every power action.

Provisioning

Setup provisioning for your environment

Namespace provisioner

The provisioner plugin adds asset provisioning capabilities to collins which can be driven via the API or the web UI.

provisioner {

  allowedStatus = [Unallocated, Maintenance, Provisioning, Provisioned]

  allowedType = [SERVER_NODE, SERVER_CHASSIS]

  enabled = false

  cacheSpecification = "expireAfterWrite=30s"

  checkCommand = "/bin/true"

  command = "/bin/false"

  profiles = "test/resources/profiles.yaml"

  rate = "1/10 seconds"

  # How long to wait for provisioning commands, checkCommand / command
  checkCommandtimeout = 40 seconds
  commandtimeout = 40 seconds
}

Provisioning Rates

Rates are expressed in terms of the number of successful requests per time period that a user can make. Specified as #/duration where # is an integer and duration is the duration. Examples:

• 1/30 seconds - once per 30 seconds
• 5/1 minute - five times per minute (same as 1/20 seconds)
• 1/0 seconds - no rate limit
• 0/0 seconds - disallowed

Substitutions in Commands

A few substitutions are available in the command and checkCommand

• <tag> - the asset tag being provisioned
• <profile-id> - the id of the profile being used for provisioning
• <notify> - the contact information supplied by the user doing the provisioning
• <suffix> - any suffix to append to the hostname as provided by the user (if allowed)
• <logfile> - a logfile to use, defaults to /tmp/<tag>-<profile-id>.log

Provisioner Profile

The provisioner profiles file is what tells collins what types of assets can be provisioned. It is a fairly simple format and leaves most actual work up to the command and check command configured CLI tools. At Tumblr we have implemented a variety of simple CLI tools to handle a variety of provisioning scenarios. Our infrastructure automation framework is called visioner, some sample visions will be distributed so you can see how some of our provisioning works.

The keys in the reference file below (databasenode, devnode, webnode) are automatically set to be the NODECLASS tag of the asset in collins. This key is also reference above as <profile-id> and is how the provision process knows what configuration to use. A profile can have any number of key/values associated with them of any type, but has a few reserved key/value pairs.

profiles:
  databasenode:
    label: "Database Server"
    prefix: "db"
    primary_role: "DATABASE"
    requires_pool: false
    contact: dbteam@company.com
    contact_notes: http://wiki.company.com/servers/databasenode
    clear_attributes:
    - TEST_DATABASE
    - DONT_WANT_THIS_ATTRIBUTE_EITHER
  devnode:
    label: "Dev Machine"
    prefix: "dev"
    allow_suffix: true
    primary_role: "DEVEL"
    pool: "DEVEL"
    attributes:
      DEV_MODE: true
  hadoopnode:
    label: "Hadoop Server"
    prefix: "hadoop"
    allowed_classes:
    - storage_class
    - service_class
  webnode:
    label: "Web Server"
    prefix: "web"
    requires_secondary_role: true

Softlayer

Working with SoftLayer

Namespace softlayer

Although Collins was initially developed as Tumblr built out its own datacenter, we found it an excellent replacement for the SoftLayer web interface. The SoftLayer plugin provides support for cancelling servers, activating servers from the spare pool, and decorating tags that are SoftLayer specific.

Note The Collins system assumes that asset tags associated with SoftLayer assets are of the form sl-ID where ID is the SoftLayer ID.

softlayer {
  enabled = true
  username = "jimbob123"
  password = "lkajsdlkjaas09823lkajdslkajsd"
  allowedCancelStatus = [Unallocated, Allocated, Maintenance]
  cancelRequestTimeout = 10 seconds
  activationRequestTimeout = 10 seconds
}

The above configuration would perform operations as user jimbob and allow you to cancel assets in the specified status.

Solr

Better search

Namespace solr

If the Solr plugin is enabled, Collins will use Solr to index assets. Since you can run Solr either embedded (bundled into Collins) or using an external instance, you should almost definitely keep this enabled.

Once the Solr plugin is enabled you will notice a few new features pop up in Collins. First, you can now query collins using the collins query language. This allows you to perform more advanced queries using ranges (how many assets have between 4 and 8 CPUs, give me assets with more than 2TB of usable storage) and much more sophisticated boolean operators. You can also sort results by any tag which is nice. You can find example queries in the collins query language section of the recipes guide.

More information about these configuration settings can be found on the official Solr documentation

Tag decorators

Taking sexy back

Namespace tagdecorators

Tag decorators allow you to change how tags are displayed on asset. Creating links, formatting names, anything is fair game. The best way to illustrate is through an example.

tagdecorators {
  templates {
    search = "<a href=\"/resources?{name}={value}\">{value}</a>"
  }
  decorators {
    APPLICATION_SHA.decorator    = "<a target=\"_blank\" href=\"https://scm.site.net/commit/{value}\">{value}</a>"
    CONTACT.decorator = "<a href=\"https://yourcompany.slack.com/messages/{value}\">{value}</a>"
    CONTACT_NOTES.decorator = "<a target=\"_blank\" href=\"{value}\">{value}</a>"
    RACK_POSITION {
      decorator = ${tagdecorators.templates.search}
      valueParser = "util.views.DelimiterParser"
      delimiter = "-"
      between = "-"
    }
    NODECLASS.decorator = ${tagdecorators.templates.search}
    POOL.decorator = ${tagdecorators.templates.search}
    PRIMARY_ROLE.decorator = ${tagdecorators.templates.search}
    SECONDARY_ROLE.decorator = ${tagdecorators.templates.search}
    BUILD_CONTACT.decorator = "<a href=\"https://yourcompany.slack.com/messages/{value}\">{value}</a>"
    IP_ADDRESS {
      decorator="<a href=\"ssh://{value}\">>{value}</a>"
      between=", "
    }
    HYPERVISOR.decorator  = "<a href=\"/resources?TAG={value}\">{value}</a>"
    AVAILABILITY_ZONE.decorator     = "<a href=\"/resources?IS_AVAILABILITY_ZONE=true&NAME={value}\">{value}</a>"
  }
}
  

Thread pools

Thread Pool

Namespace akka

Configuring Collins akka pools is relatively straightforward. See the akka documentation for details. Collins makes use of a single akka dispatcher which can be configured in akka.actor.default-dispatcher. The dispatcher is shared for background processes and solr updates

Note Currently only on the development branch, scheduled for future release.

akka {
  actor {
    default-dispatcher = {
      fork-join-executor {
        parallelism-factor = 1.0
      }
    }

    deployment = {
      /background-processor = {
        dispatcher = default-dispatcher
        router = round-robin
        nr-of-instances = 128
      }

      /solr_asset_updater = {
        dispatcher = default-dispatcher
      }

      /solr_asset_log_updater = {
        dispatcher = default-dispatcher
      }
    }
  }
}