Prom-migrator is an open-source, community-driven and free-to-use, universal prometheus data migration tool, that migrates data from one storage system to another, leveraging Prometheus's remote storage endpoints.
- Migrate prometheus data from one storage system to another
- Pretty outputs during runtime, allowing users to track progress
- Ability to resume migration(s) from last sent data, in case of any unintended shutdowns
- Stateless working model
This document contains:
- Working overview, that describes in brief, the working model of prom-migrator
- Storage systems
- CLI flags
- Contributing
- Help
Prom-migrator migrates the data from one storage to another. It pulls the data from a particular remote storage system using the remote-read endpoint for a certain time-range. It then pushes the data to another remote storage system using the remote-write endpoint. It continues by reading and then writing more data until it finishes the entire time-range specified by the user. The time-range of any individual read is adaptively auto-adjusted to bound the overall memory usage of the system.
The system is able to auto-resume the migration if a previous migration was interrupted. It does this by adding a progress-metric to the data as it is writing. The sample of the progress metric records the maximum time that was written. Thus, when a migration process resumes, it simply reads the progress-metric to find out what was last written and picks up where it left off.
For detailed information about working, design and process, refer to https://tsdb.co/prom-migrator-design-doc
Storage systems that we aim to support with prom-migrator are:
Key:
- read - means data can be read from system,
- write - means data can be written to an empty database,
- backfill - means data can be written to a database that already contains data newer than what is being inserted.
- Promscale (read, write, backfill)
- Prometheus (read)
- Prometheus tsdb (read, TODO in next version)
- Thanos (read, write)
- Cortex (only blocks storage, chunks storage in later versions) (read, write)
- VictoriaMetrics (read, write, not sure about backfill)
- M3DB (read, write, not sure about backfill)
- Others (any remote storage systems with remote read (for reading) and write (for writing) endpoints)
It should be noted that some of the above systems do not support backfill and so their ability to serve as write endpoints are somewhat constrained.
./prom-migrator -mint=1606408552 -maxt=1606415752 -read-url=<read_endpoint_url_for_remote_read_storage> -write-url=<write_endpoint_url_for_remote_write_storage> -writer-read-url=<read_endpoint_url_for_remote_write_storage>
| Flag | Type | Required | Default | Description |
|---|---|---|---|---|
| mint | int64 | true | 0 | Minimum timestamp (in seconds unix) for carrying out data migration. (inclusive) |
| read-url | string | true | URL address for the storage where the data is to be read from. | |
| write-url | string | true | URL address for the storage where the data migration is to be written. | |
| maxt | int64 | false | time.Now().Unix() | Maximum timestamp (in seconds unix) for carrying out data migration (exclusive). Setting this value less than zero will indicate all data from mint upto now. |
| migration-name | string | false | prom-migrator | Name for the current migration that is to be carried out. It corresponds to the value of the label 'job' set inside the progress-metric-name. |
| max-read-size | string | false | 500MB | (units: B, KB, MB, GB, TB, PB) the maximum size of data that should be read at a single time. More the read size, faster will be the migration but higher will be the memory usage. Example: 250MB. |
| progress-enabled | boolean | false | true | This flag tells the migrator, whether or not to use the progress mechanism. It is helpful if you want to carry out migration with the same time-range. If this is enabled, the migrator will resume the migration from the last time, where it was stopped/interrupted. If you do not want any extra metric(s) while migration, you can set this to false. But, setting this to false will disble progress-metric and hence, the ability to resume migration. |
| progress-metric-name | string | false | prom_migrator_progress | Prometheus metric name for tracking thelast maximum timestamp pushed to the remote-write storage. This is used to resume the migration process after a failure. |
| progress-metric-url | string | false | URL of the remote storage that contains the progress-metric. Note: This url is used to fetch the last pushed timestamp. If you want the migration to resume from where it left, in case of a crash, set this to the remote write storage that the migrator is writing along with the progress-enabled. |
Note: A simple way to find timestamp in seconds unix, you can simply query in prometheus's (or related platform) UI and see the start/end timestamp. If in decimal, the part prior to the decimal point will be the timestamp in seconds unix.
| Flag | Type | Description |
|---|---|---|
| read-auth-username | string | Auth username for remote-read storage. |
| read-auth-password | string | Auth password for remote-read storage. |
| read-auth-bearer-token | string | Bearer token for remote-read storage. This should be mutually exclusive with username and password. |
| Flag | Type | Description |
|---|---|---|
| write-auth-username | string | Auth username for remote-write storage. |
| write-auth-password | string | Auth password for remote-write storage. |
| write-auth-bearer-token | string | Bearer token for remote-write storage. This should be mutually exclusive with username and password. |
Note: Currently, credentials are expected to be supplied via CLI. In later versions, reading credentials from files (recommended for security reasons than providing credentials through CLI) will be supported.
We welcome contributions to the Promscale Connector, which is licensed and released under the open-source Apache License, Version 2. The same Contributor's Agreement applies as in TimescaleDB; please sign the Contributor License Agreement (CLA) if you're a new contributor.
In case of any issues/bugs/support, please feel free to open github issues at the promscale repo issues page.
For additional details/queries/feedback/support, please feel free to ask us on the #promscale channel on TimescaleDB Slack. You can also email us at [email protected] or meet us virtually by joining the promscale community call, which is held on second wednesday of every month on zoom (link) (agenda link) or open topics on the required help at Promscale Users Google Group.