v0.13.0

go-ipfs v0.13.0 Release

We're happy to announce go-ipfs 0.13.0, packed full of changes and improvements!

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

Overview

Below is an outline of all that is in this release, so you get a sense of all that's included.

• 🛠 BREAKING CHANGES
  • ipfs block put command
  • ipfs cid codecs command
  • ipfs cid format command
  • Swarm configuration
  • Circuit Relay V1 is deprecated
  • ls requests for /multistream/1.0.0 are removed
  • Gateway Behavior
• 🔦 Highlights
  • 🧑‍💼 libp2p Network Resource Manager (Swarm.ResourceMgr)
  • 🔃 Relay V2 client with auto discovery (Swarm.RelayClient)
  • 🌉 HTTP Gateway improvements
    • 🍱 Support for Block and CAR response formats
    • 🐎 Fast listing generation for huge directories
    • 🎫 Improved Etag and If-None-Match for bandwidth savings
    • ⛓️ Added X-Ipfs-Roots for smarter HTTP caches
    • 🌡️ Added metrics per response type
  • 🕵️ OpenTelemetry tracing
    • How to use Jaeger UI for visual tracing?
  • 🩺 Built-in ipfs diag profile to ease debugging
  • 🔑 Support for PEM/PKCS8 for key import/export
  • 🧹 Using standard IPLD codec names across the CLI/HTTP API
  • 🐳 Custom initialization for Docker
  • RPC API docs for experimental and deprecated commands
  • Yamux over Mplex

🛠 BREAKING CHANGES

ipfs block put command

ipfs block put command returns a CIDv1 with raw codec by default now.

• ipfs block put --cid-codec makes block put return CID with alternative codec
  • This impacts only the returned CID; it does not trigger any validation or data transformation.
  • Retrieving a block with a different codec or CID version than it was put with is valid.
  • Codec names are validated against tables from go-multicodec library.
• ipfs block put --format is deprecated. It used incorrect codec names and should be avoided for new deployments. Use it only if you need the old, invalid behavior, namely:
  • ipfs block put --format=v0 will produce CIDv0 (implicit dag-pb)
  • ipfs block put --format=cbor will produce CIDv1 with dag-cbor (!)
  • ipfs block put --format=protobuf will produce CIDv1 with dag-pb (!)

ipfs cid codecs command

• Now lists codecs from go-multicodec library.
• ipfs cid codecs --supported can be passed to only show codecs supported in various go-ipfs commands.

ipfs cid format command

• --codec was removed and replaced with --mc to ensure existing users are aware of the following changes:
  • --mc protobuf now correctly points to code 0x50 (was 0x70, which is dab-pg)
  • --mc cbor now correctly points to code 0x51 (was 0x71, which is dag-cbor)

Swarm configuration

• Daemon will refuse to start if long-deprecated RelayV1 config key Swarm.EnableAutoRelay or Swarm.DisableRelay is set to true.
• If Swarm.Transports.Network.Relay is disabled, then Swarm.RelayService and Swarm.RelayClient are also disabled (unless they have been explicitly enabled).

Circuit Relay V1 is deprecated

• By default, Swarm.RelayClient does not use Circuit Relay V1. Circuit V1 support is only enabled when Swarm.RelayClient.StaticRelays are specified.

ls requests for /multistream/1.0.0 are removed

• go-libp2p 0.19 removed support for undocumented ls command (PR). If you are still using it for internal testing, it is time to refactor (example)

Gateway Behavior

Directory listings returned by the HTTP Gateway won't have size column if the directory is bigger than Gateway.FastDirIndexThreshold config (default is 100).

To understand the wider context why we made these changes, read Highlights below.

🔦 Highlights

🧑‍💼 libp2p Network Resource Manager (Swarm.ResourceMgr)

You can now easily bound how much resource usage libp2p consumes! This aids in protecting nodes from consuming more resources then are available to them.

The libp2p Network Resource Manager is disabled by default, but can be enabled via:

ipfs config --json Swarm.ResourceMgr.Enabled true

When enabled, it applies some safe defaults that can be inspected and adjusted with:

• ipfs swarm stats --help
• ipfs swarm limit --help

User changes persist to config at Swarm.ResourceMgr.

The Resource Manager will be enabled by default in a future release.

🔃 Relay V2 client with auto discovery (Swarm.RelayClient)

All the pieces are enabled for hole-punching by default, improving connecting with nodes behind NATs and Firewalls!

This release enables Swarm.RelayClient by default, along with circuit v2 relay discovery provided by go-libp2p v0.19.0. This means:

1. go-ipfs will coordinate with the counterparty using a relayed connection, to upgrade to a direct connection through a NAT/firewall whenever possible.
2. go-ipfs daemon will automatically use public relays if it detects that it cannot be reached from the public internet (e.g., it's behind a firewall). This results in a /p2p-circuit address from a public relay.

Notes:

• Swarm.RelayClient does not use Circuit Relay V1 nodes any more. Circuit V1 support is only enabled when static relays are specified in Swarm.RelayClient.StaticRelays.
• One can opt-out via Swarm.EnableHolePunching.

🌉 HTTP Gateway improvements

HTTP Gateway enables seamless interop with the existing Web, clients, user agents, tools, frameworks and libraries.

This release ships the first batch of improvements that enable creation of faster and smarter CDNs, and unblocks creation of light clients for Mobile and IoT.

Details below.

🍱 Support for Block and CAR response formats

Alternative response formats from Gateway can be requested to avoid needing to trust a gateway.

For now, {format} is limited to two options:

• raw – fetching single block
• car – fetching entire DAG behind a CID as a CARv1 stream

When not set, the default UnixFS response is returned.

Why these two formats? Requesting Block or CAR for /ipfs/{cid} allows a client to use gateways in a trustless fashion. These types of gateway responses can be verified locally and rejected if digest inside of requested CID does not match received bytes. This enables creation of "light IPFS clients" which use HTTP Gateways as inexpensive transport for content-addressed data, unlocking use in Mobile and IoT contexts.

Future releases will add support for dag-json and dag-cbor responses.

There are two ways for requesting CID specific response format:

1. HTTP header: Accept: application/vnd.ipld.{format}

• Examples: application/vnd.ipld.car, application/vnd.ipld.raw

2. URL paramerer: ?format=

• Useful for creating "Download CAR" links.

Usage examples:

1. Downloading a single raw Block and manually importing it to the local datastore:

$ curl  -H 'Accept: application/vnd.ipld.raw' "http://127.0.0.1:8080/ipfs/QmZULkCELmmk5XNfCgTnCyFgAVxBRBXyDHGGMVoLFLiXEN" --output block.bin
...

v0.13.0-rc1

Tracking Issue: #8640

v0.12.2

go-ipfs v0.12.2 Release

This patch release fixes a security issue wherein traversing some malformed DAGs can cause the node to panic.

Given that some users haven't yet gone through the migration for v0.12.0, we have also backported this to v0.11.1.

See also the security advisory: GHSA-mcq2-w56r-5w2w

Changelog

Full Changelog

- github.com/ipld/go-codec-dagpb (v1.3.0 -> v1.3.2): - fix: use protowire for Links bytes decoding

❤ Contributors

Contributor	Commits	Lines ±	Files Changed
Rod Vagg	1	+34/-19	2

v0.11.1

go-ipfs v0.11.1 Release

This patch release covers a couple of security fixes

Malformed DAG Traversal

This patch release fixes a security issue wherein traversing some malformed DAGs can cause the node to panic.

This was backported from v0.12.2, since some users haven't yet gone through the v0.12 migration.

See also the security advisory: GHSA-mcq2-w56r-5w2w

Docker Compose Ports

This patch release fixes a security issue with the docker-compose.yaml file in which the IPFS daemon API listens on all interfaces instead of only the loopback interface, which could allow remote callers to control your IPFS daemon. If you use the included docker-compose.yaml file, it is recommended to upgrade.

See also the security advisory: GHSA-fx5p-f64h-93xc

Thanks to @LynHyper for finding and disclosing this.

Changelog

Full Changelog

- github.com/ipfs/go-ipfs: - fix: listen on loopback for API and gateway ports in docker-compose.yaml - github.com/ipld/go-codec-dagpb (v1.3.0 -> v1.3.2): - fix: use protowire for Links bytes decoding

❤ Contributors

Contributor	Commits	Lines ±	Files Changed
Rod Vagg	1	+34/-19	2
guseggert	1	+10/-3	1

v0.12.1

go-ipfs v0.12.1 Release

This patch release fixes a security issue with the docker-compose.yaml file in which the IPFS daemon API listens on all interfaces instead of only the loopback interface, which could allow remote callers to control your IPFS daemon. If you use the included docker-compose.yaml file, it is recommended to upgrade.

See also the security advisory: GHSA-fx5p-f64h-93xc

Thanks to @LynHyper for finding and disclosing this.

Changelog

Full Changelog

• github.com/ipfs/go-ipfs:
  • fix: listen on loopback for API and gateway ports in docker-compose.yaml

❤ Contributors

Contributor	Commits	Lines ±	Files Changed
guseggert	1	+10/-3	1

v0.12.0

go-ipfs 0.12.0 Release

We're happy to announce go-ipfs 0.12.0. This release switches the storage of IPLD blocks to be keyed by multihash instead of CID.

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

🛠 BREAKING CHANGES

• ipfs refs local will now list all blocks as if they were raw CIDv1 instead of with whatever CID version and IPLD codecs they were stored with. All other functionality should remain the same.

Note: This change also effects ipfs-update so if you use that tool to mange your go-ipfs installation then grab ipfs-update v1.8.0 from dist.

Keep reading to learn more details.

🔦 Highlights

There is only one change since 0.11:

Blockstore migration from full CID to Multihash keys

We are switching the default low level datastore to be keyed only by the Multihash part of the CID, and deduplicate some blocks in the process. The blockstore will become codec-agnostic.

Rationale

The blockstore/datastore layers are not concerned with data interpretation, only with storage of binary blocks and verification that the Multihash they are addressed with (which comes from the CID), matches the block. In fact, different CIDs, with different codecs prefixes, may be carrying the same multihash, and referencing the same block. Carrying the CID abstraction so low on the stack means potentially fetching and storing the same blocks multiple times just because they are referenced by different CIDs. Prior to this change, a CIDv1 with a dag-cbor codec and a CIDv1 with a raw codec, both containing the same multihash, would result in two identical blocks stored. A CIDv0 and CIDv1 both being the same dag-pb block would also result in two copies.

How migration works

In order to perform the switch, and start referencing all blocks by their multihash, a migration will occur on update. This migration will take the repository version from 11 (current) to 12.

One thing to note is that any content addressed CIDv0 (all the hashes that start with Qm..., the current default in go-ipfs), does not need any migration, as CIDv0 are raw multihashes already. This means the migration will be very lightweight for the majority of users.

The migration process will take care of re-keying any CIDv1 block so that it is only addressed by its multihash. Large nodes with lots of CIDv1-addressed content will need to go through a heavier process as the migration happens. This is how the migration works:

1. Phase 1: The migration script will perform a pass for every block in the datastore and will add all CIDv1s found to a file named 11-to-12-cids.txt, in the go-ipfs configuration folder. Nothing is written in this first phase and it only serves to identify keys that will be migrated in phase 2.
2. Phase 2: The migration script will perform a second pass where every CIDv1 block will be read and re-written with its raw-multihash as key. There is 1 worker performing this task, although more can be configured. Every 100MiB-worth of blocks (this is configurable), each worker will trigger a datastore "sync" (to ensure all written data is flushed to disk) and delete the CIDv1-addressed blocks that were just renamed. This provides a good compromise between speed and resources needed to run the migration.

At every sync, the migration emits a log message showing how many blocks need to be rewritten and how far the process is.

FlatFS specific migration

For those using a single FlatFS datastore as their backing blockstore (i.e. the default behavior), the migration (but not reversion) will take advantage of the ability to easily move/rename the blocks to improve migration performance.

Unfortunately, other common datastores do not support renames which is what makes this FlatFS specific. If you are running a large custom datastore that supports renames you may want to consider running a fork of fs-repo-11-to-12 specific to your datastore.

If you want to disable this behavior, set the environment variable IPFS_FS_MIGRATION_11_TO_12_ENABLE_FLATFS_FASTPATH to false.

Migration configuration

For those who want to tune the migration more precisely for their setups, there are two environment variables to configure:

• IPFS_FS_MIGRATION_11_TO_12_NWORKERS : an integer describing the number of migration workers - defaults to 1
• IPFS_FS_MIGRATION_11_TO_12_SYNC_SIZE_BYTES : an integer describing the number of bytes after which migration workers will sync - defaults to 104857600 (i.e. 100MiB)

Migration caveats

Large repositories with very large numbers of CIDv1s should be mindful of the migration process:

• We recommend ensuring that IPFS runs with an appropriate (high) file-descriptor limit, particularly when Badger is use as datastore backend. Badger is known to open many tables when experiencing a high number of writes, which may trigger "too many files open" type of errors during the migrations. If this happens, the migration can be retried with a higher FD limit (see below).
• Migrations using the Badger datastore may not immediately reclaim the space freed by the deletion of migrated blocks, thus space requirements may grow considerably. A periodic Badger-GC is run every 2 minutes, which will reclaim space used by deleted and de-duplicated blocks. The last portion of the space will only be reclaimed after go-ipfs starts (the Badger-GC cycle will trigger after 15 minutes).
• While there is a revert process detailed below, we recommend keeping a backup of the repository, particularly for very large ones, in case an issue happens, so that the revert can happen immediately and cases of repository corruption due to crashes or unexpected circumstances are not catastrophic.

Migration interruptions and retries

If a problem occurs during the migration, it is be possible to simply re-start and retry it:

1. Phase 1 will never overwrite the 11-to-12-cids.txt file, but only append to it (so that a list of things we were supposed to have migrated during our first attempt is not lost - this is important for reverts, see below).
2. Phase 2 will proceed to continue re-keying blocks that were not re-keyed during previous attempts.

Migration reverts

It is also possible to revert the migration after it has succeeded, for example to go to a previous go-ipfs version (<=0.11), even after starting and using go-ipfs in the new version (>=0.12). The revert process works as follows:

1. The 11-to-12-cids.txt file is read, which has the list of all the CIDv1s that had to be rewritten for the migration.
2. A CIDv1-addressed block is written for every item on the list. This work is performed by 1 worker (configurable), syncing every 100MiB (configurable).
3. It is ensured that every CIDv1 pin, and every CIDv1 reference in MFS, are also written as CIDV1-addressed blocks, regardless of whether they were part of the original migration or were added later.

The revert process does not delete any blocks--it only makes sure that blocks that were accessible with CIDv1s before the migration are again keyed with CIDv1s. This may result in a datastore becoming twice as large (i.e. if all the blocks were CIDv1-addressed before the migration). This is however done this way to cover corner cases: user can add CIDv1s after migration, which may reference blocks that existed as CIDv0 before migration. The revert aims to ensure that no data becomes unavailable on downgrade.

While go-ipfs will auto-run the migration for you, it will not run the reversion. To do so you can download the latest migration binary or use ipfs-update.

Custom datastores

As with previous migrations if you work with custom datastores and want to leverage the migration you can run a fork of fs-repo-11-to-12 specific to your datastore. The repo includes instructions on building for different datastores.

For this migration, if your datastore has fast renames you may want to consider writing some code to leverage the particular efficiencies of your datastore similar to what was done for FlatFS.

Changelog

Full Changelog

• github.com/ipfs/go-ipfs:
  • Release v0.12.0
  • docs: v0.12.0 release notes
  • chore: bump migrations dist.ipfs.io CID to contain fs-repo-11-to-12 v1.0.2
  • feat: refactor Fetcher interface used for downloading migrations (#8728) (ipfs/go-ipfs#8728)
  • feat: log multifetcher errors
  • Release v0.12.0-rc1
  • chore: bump Go version to 1.16.12
  • feat: switch to raw multihashes for blocks (ipfs/go-ipfs#6816)
  • chore: add release template snippet for fetching artifact tarball
  • chore: bump Go version to 1.16.11
  • chore: add release steps for upgrading Go
  • Merge branch 'release'
  • fix(corehttp): adjust peer counting metrics (#8577) (ipfs/go-ipfs#8577)
  • chore: update version to v0.12.0-dev
• github.com/ipfs/go-filestore (v0.1.0 -> v1.1...

v0.12.0-rc1

Tracking Issue: #8344

v0.11.0

go-ipfs 0.11.0 Release

We're happy to announce go-ipfs 0.11.0. This release comes with improvements to the UnixFS Sharding and PubSub experiments as well as support for Circuit-Relay v2 which sets the network up for decentralized hole punching support.

As usual, this release includes important fixes, some of which may be critical for security. Unless the fix addresses a bug being exploited in the wild, the fix will not be called out in the release notes. Please make sure to update ASAP. See our release process for details.

🛠 BREAKING CHANGES

• UnixFS sharding is now automatic and enabled by default
  • HAMT-based sharding is applied to large directories (i.e. those that would serialize into block larger than ~256KiB)s. This means importing data via commands like ipfs add -r <directory> may result in different CIDs due to the different DAG representations.
  • Support for Experimental.ShardingEnabled is removed.
• go-ipfs can no longer act as a Circuit Relay v1
  • Node will refuse to start if Swarm.EnableRelayHop is set to true
  • If you depend on v1 relay service provider, see "Removal of v1 relay service" section for available migration options.
• HTTP RPC wire format for experimental commands at /api/v0/pubsub changed.
  • If you use js-ipfs-http-client or go-ipfs-http-client, just update to their latest version.
  • If you use something else, see "Multibase in PubSub" section below for migration details.

Keep reading to learn more details.

🔦 Highlights

🗃️ Automatic UnixFS sharding

Truly big directories can have so many items, that the root block with all of their names is too big to be exchanged with other peers. This was partially solved by HAMT-sharding, which was introduced a while ago as opt-in. The main downside of the implementation was that it was a global flag that sharded all imported directories (big and small).

This release solves that inconvenience by making UnixFS sharding smarter and applies it only to larger directories (i.e. directories that would be at least ~256KiB). This is now the default behavior in ipfs add and ipfs files commands, where UnixFS sharding works out-of-the-box.

🔁 Circuit Relay v2

This release adds support for the circuit relay v2 protocol based on the reference implementation from go-libp2p 0.16.

This is the cornerstone for maximizing p2p connections between IPFS peers. Every publicly dialable peer can now act as a limited relay v2, which can be used for hole punching and other decentralized signaling protocols.

Limited relay v2 configuration options

go-ipfs can now be configured to act as a RelayClient that uses other peers for autorelay functionality when behind a NAT, or provide a limited RelayService to other peers on the network.

Starting with go-ipfs v0.11 every publicly dialable go-ipfs (based on AutoNAT determination) will start a limited RelayService. RelayClient remains disabled by default for now, as we want the network to update and get enough v2 service providers first.

Note: the limited Circuit Relay v2 provided with this release only allows low-bandwidth protocols (identify, ping, holepunch) over transient connections. If you want to relay things like bitswap sessions, you need to set up a v1 relay by some other means. See details below.

Removal of unlimited v1 relay service provider

Switching to v2 of the relay spec means removal or deprecation of configuration keys that were specific to v1.

• Relay transport and client support circuit-relay v2:
  • Swarm.EnableAutoRelay was replaced by Swarm.RelayClient.Enable.
  • Swarm.DisableRelay is deprecated, relay transport can be now disabled globally (both client and service) by setting Swarm.Transports.Network.Relay to false
• Relay v1 service provider was replaced by v2:
  • Swarm.EnableRelayHop no longer starts an unlimited v1 relay. If you have it set to true the node will refuse to start and display an error message.
  • Existing users who choose to continue running a v1 relay should migrate their setups to relay v1 based on js-ipfs running in node, or the standalone libp2p-relay-daemon configured with RelayV1.Enabled set to true. Be mindful that v1 relays are unlimited, and one may want to set up some ACL based either on PeerIDs or Subnets.

🕳 Decentralized Hole Punching (DCUtR protocol client)

We are working towards enabling hole punching for NAT traversal when port forwarding is not possible.

go-libp2p 0.16 provides an implementation of the DCUtR (decentralized hole punching) protocol. It is hidden behind the Swarm.EnableHolePunching configuration flag.

When enabled, go-ipfs will coordinate with the counterparty using a relayed v2 connection, to upgrade to a direct connection through a NAT/firewall whenever possible.

This feature is disabled by default in this release, but we hope to enable it by default as soon the network updates to go-ipfs v0.11 and gains a healthy set of limited v2 relays.

💬 Multibase in PubSub HTTP RPC API

This release fixed some edge cases that were reported by users of the PubSub experiment, getting it closer to becoming a stable feature of go-ipfs. Some PubSub users will notice that the plaintext limitation is lifted: one can now use line breaks in messages published to non-ascii topic names, or even publish arbitrary bytes to arbitrary topics. It required a change to the wire format used when pubsub commands are executed over the HTTP RPC API at /api/v0/pubsub/*, and also modified the behavior of the ipfs pubsub pub command, which now is publishing only a single pubsub message with data read from a file or stdin.

PubSub client migration tips

If you use the HTTP RPC API with the go-ipfs-http-client library, make sure to update to the latest version. The next version of js-ipfs-http-client will use the new wire format as well, so you don't need to do anything.

If you use /api/v0/pubsub/* directly or maintain your own client library, you must adjust your HTTP client code. Byte fields and URL args are now encoded in base64url Multibase. Encode/decode bytes using the ipfs multibase --help commands, or use the multiformats libraries (js-multiformats, go-multibase).

Low level changes:

• topic passed as URL arg in requests to /api/v0/pubsub/* must be encoded in URL-safe multibase (base64url)
• data, from, seqno and topicIDs returned in JSON responses are now encoded in multibase
• Peer IDs returned in from now use the same default text representation from go-libp2p and peerid encoder/decoder from libp2p. This means the same text representation as in as in swarm peers, which makes it possible to compare them without decoding multibase.
• /api/v0/pubsub/pub no longer accepts data to be passed as URL, it has to be sent as multipart/form-data. This removes size limitations based on URL length, and enables regular HTTP clients to publish data to PubSub topics. For example, to publish some-file to topic named test-topic using vanilla curl, one would execute: curl -X POST -v -F "stdin=@some-file" 'http://127.0.0.1:5001/api/v0/pubsub/pub?arg=$(echo -n test-topic | ipfs multibase encode -b base64url)'
• ipfs pubsub pub on the command line no longer accepts variadic data arguments. Instead, it expects a single file input or stream of bytes from stdin. This ensures arbitrary stream of bytes can be published, removing limitation around messages that include \n or \r\n.

⚙️ New configuration flags

• Addresses.AppendAnnounce is an array of multiaddrs, similar to Addresses.Announce, except it does not override inferred swarm addresses, but appends custom ones to the list.
• Pubsub experiments can now be enabled via config, removing the need for CLI flag to be passed every time daemon starts:
  • Pubsub.Enabled enables the pubsub system.
  • Ipns.UsePubsub enables IPFS over pubsub experiment for publishing IPNS records in real time.

🔐 Support for DAG-JOSE IPLD codec

JOSE is a standard for signing and encrypting JSON objects. DAG-JOSE is an IPLD codec based on JO...

v0.11.0-rc2

Tracking Issue: #8343

v0.11.0-rc1

Tracking Issue: #8343