influxdb: upgrade to 0.10 considerations

Upgrading to InfluxDB 0.10: Things to ConsiderSean Beckett Director of Support & Professional Services

• New Storage Engine, TSM • New Continuous Query syntax • New Clustering

What's New?

0.8 —> 0.10 • New Schema • New API and endpoints • No MERGEs • Shard Spaces become Retention Policies • Updated Query syntax • Administrative queries replace Admin UI

• Clustering in InfluxDB 0.10 is not backwards compatible

• Continuous Query syntax and configuration are different

Brief Mentions

https://github.com/influxdata/influxdb/blob/master/importer/README.md https://docs.influxdata.com/influxdb/v0.9/concepts/08_vs_09/ https://docs.influxdata.com/influxdb/v0.10/concepts/09_vs_010/

TSM: Time Structured Merge-Tree

• Purpose-built • Compression by design • No "empty" space • Read only what is needed

TSM: Best Practices - measurements

• Measurements are logical containers for series • Measurements contain series with related field sets • No JOINs or MERGEs across measurements

TSM: Best Practices - tags

• Tags are indexed • Tags in the WHERE clause are performant • GROUP BY operates on tags • Tag Cardinality matters!

TSM: Best Practices - fields

• Fields are not indexed • Fields in the WHERE clause are much less performant • Functions operate only on fields • Field cardinality is irrelevant* • Multiple fields per point is more performant

TSM: Best Practices - timestamps

• Timestamp precision is controlled at write time • Lower precision uses less disk, faster writes and reads

Migration Tools

InfluxDB 0.8.9 - Export (endpoint) InfluxDB 0.9+ - Import via CLI (./influx) InfluxDB 0.10+ - TSM conversion tool (./influx_tsm)

Example Migration #1

Conditions: • Downtime is irrelevant • Converting from 0.9 • Not clustered


1. Stop traffic to InfluxDB 0.9 2. Restart the process and wait for the WAL to flush 3. Stop InfluxDB 4. Install 0.10.1 on top of 0.9 5. Run ./influx_tsm to convert shards

influx_tsm -parallel /var/lib/influxdb/data

6. Once complete, launch InfluxDB 0.10.1 7. Resume traffic to InfluxDB


Conditions: • Downtime is unacceptable • Converting from 0.9 • Not clustered


1. Install 0.10.1 on top of 0.9 and restart 2. Copy "cold" B1/BZ1 shards to /tmp 3. Run ./influx_tsm to convert cold shards

influx_tsm -backup /tmp/influxdb_backup /tmp/data

4. Remove cold B1/BZ1 shards and replace with converted TSM shards

5. Wait until current "hot" shards become "cold", repeat steps 2-4 on newly cold shards


Conditions: • Downtime is acceptable • Converting from 0.8 • Not clustered


1. Stop traffic to InfluxDB 0.8.x 2. Install InfluxDB 0.8.9 and restart 3. Run the Export tool

curl -o export.gz --compressed http://localhost:8086/ \ export/<db>/<shard_space>

4. Launch InfluxDB 0.10.1 on a new server 5. Use the CLI to import the exported data

influx -import -path=export.gz -compressed > failures

6. Update clients and dashboards for new API, etc. 7. Resume traffic to InfluxDB 0.10.1

The Hard Stuff

• Fixing failed points in 0.8 import • Migrating 0.8 queries where a column is treated as

both a tag and a field • Controlling series cardinality • Updating the clients to avoid downtime • Updating the dashboards to avoid downtime • Controlling for performance changes

Questions

docs.influxdata.com [email protected] [email protected]

influxdb: upgrade to 0.10 considerations

Data & Analytics