{"@context":"https://schema.org","@type":"Article","headline":"Infrastructure Upgrades","image":"https://massive.com/blog/content/images/2018/01/infra-upgrades.jpg","description":"Over the past 2 weeks we have made several infrastructure changes and upgrades.\nThe changes touched almost every part of the system, therefor you may have\nnoticed breif latency spikes up to 200ms. However we tried to keep the upgrades\ncompletely transparent.\n\nThe upgrades allow for us to add real-time data feeds more easily, with less\nlatency and better message delivery guarantees. Which means better APIs and more\ndata streams for you.\n\nBelow are some details for the techies who are interested.\n","publisher":{"@type":"Organization","name":"Massive","url":"https://massive.com/blog","logo":{"@type":"ImageObject","url":"https://massive.com$/brands/logo-massive.png"}},"url":"https://massive.com/blog/infrastructure-upgrades/","keywords":["infrastructure","kafka","ceph"],"datePublished":"2018-01-01T20:25:00.000-05:00","dateCreated":"2018-01-08T19:45:07.000-05:00","dateModified":"2018-01-09T01:00:04.000-05:00","articleBody":"

Over the past 2 weeks we have made several infrastructure changes and upgrades. The changes touched almost every part of the system, therefor you may have noticed breif latency spikes up to 200ms. However we tried to keep the upgrades completely transparent.

The upgrades allow for us to add real-time data feeds more easily, with less latency and better message delivery guarantees. Which means better APIs and more data streams for you.

Below are some details for the techies who are interested.

\n $\"Ceph$ \n

Ceph Luminous

One of the upgrades was to the Ceph cluster which brings it to the latest version, allowing us to mark the SSD OSDs and SAS OSDs accordingly. Along with marking the OSDs it's now easy to specify which data should live on those lables. This means our DBs and critical data can live on the SSDs only while logs and non-latency critical data can live on the SAS drives. Snapshots and archive data can live on traditional SATAs.

Along with the upgrade, we have added additional monitoring to prometheus for each storage pool and node. Giving better insight if a single node or OSD is having slow read/write speeds.

We have also converted several of our OSDs to use the new BlueStore filesystem format. ( Read more about BlueStore here ). This eliminates the need of an SSD for the OSD log file, freeing up more disks for usable storage. If after a couple months these show promising results, we will continue switch more to this format.

\n $\"Apache$ \n

Apache Kafka

We have switched our internal messaging system to a 16 node Kafka cluster. Taking advantage of the streaming mechanisms for data manipulation and writing to C*. This gives us several benefits including data durability, replication, and splitting up streams between groups of services. With splitting streams between multiple instances of an application our uptime will not suffer due to a server outage. We can now scale our infrastructure horizontally as needed. Currently we can handle 2m+ ticks/second with only 1-2ms latency.

While we normally try to steer away from JVM based projects, the performance and proven reliability of Kafka is hard to beat. The streaming landscape is getting a lot of attention right now, so it will be interesting to see what projects emerge.

\n","author":{"@type":"Person","name":"Quinton Pike","image":{"@type":"ImageObject","url":"https://massive.com/blog/content/images/2018/01/datacenter-1.jpg"},"url":"https://massive.com/blog/author/quintonpike/"},"mainEntityOfPage":"https://massive.com/blog/infrastructure-upgrades/"}