Skip to content

Commit

Permalink
docs: cleanups for the rename, fixing links (#1624)
Browse files Browse the repository at this point in the history
  • Loading branch information
MingweiSamuel authored Dec 22, 2024
1 parent 73aee2c commit 6ab6252
Show file tree
Hide file tree
Showing 32 changed files with 130 additions and 137 deletions.
9 changes: 4 additions & 5 deletions .github/gh-pages/index.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
---
---
- [The Hydroflow Book]({{ "/book/" | prepend: site.github.url | replace: 'http://', '//' }})
- [Rust Docs]({{ "/doc/hydroflow/" | prepend: site.github.url | replace: 'http://', '//' }})
- [Design Docs]({{ "/design_docs/" | prepend: site.github.url | replace: 'http://', '//' }})
- [Benchmark History]({{ "/bench/" | prepend: site.github.url | replace: 'http://', '//' }})
- [Latest Benchmarks]({{ "/criterion/report/" | prepend: site.github.url | replace: 'http://', '//' }})
- [Main website (hydro.run) 🔗](https://hydro.run)
- [Github Repository 🔗](https://github.com/hydro-project/hydroflow)
- [Design Docs]({{ "/design_docs/" | prepend: site.github.url | replace: 'https://', '//' }})
- [Benchmark History]({{ "/bench/" | prepend: site.github.url | replace: 'https://', '//' }})
- [Latest Benchmarks]({{ "/criterion/report/" | prepend: site.github.url | replace: 'https://', '//' }})
4 changes: 2 additions & 2 deletions .github/workflows/template.yml
Original file line number Diff line number Diff line change
Expand Up @@ -84,8 +84,8 @@ jobs:
echo "$ECHO_STRING" >client-input
"${{ runner.temp }}/generated/target/debug/hydroflow-template" --role server --address 127.0.0.100:2048 &
"${{ runner.temp }}/generated/target/debug/hydroflow-template" --role client --address 127.0.0.100:2048 <client-input >client-output &
"${{ runner.temp }}/generated/target/debug/dfir-template" --role server --address 127.0.0.100:2048 &
"${{ runner.temp }}/generated/target/debug/dfir-template" --role client --address 127.0.0.100:2048 <client-input >client-output &
sleep 1
Expand Down
34 changes: 12 additions & 22 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
# Visit https://bit.ly/cffinit to generate yours today!

cff-version: 1.2.0
title: Hydroflow
title: Hydro
message: >-
If you use this software, please cite it using the
metadata from this file.
Expand All @@ -13,10 +13,6 @@ authors:
email: [email protected]
affiliation: Sutter Hill Ventures
orcid: 'https://orcid.org/0009-0004-9873-6266'
- given-names: Justin
family-names: Jaffray
email: [email protected]
affiliation: Sutter Hill Ventures
- given-names: Shadaj
family-names: Laddad
email: [email protected]
Expand All @@ -27,17 +23,18 @@ authors:
email: [email protected]
orcid: 'https://orcid.org/0000-0002-7712-4306'
affiliation: UC Berkeley
- given-names: Rohit
family-names: Kulshreshtha
email: [email protected]
affiliation: Sutter Hill Ventures
- given-names: Lucky
family-names: Katahanas
email: [email protected]
affiliation: Sutter Hill Ventures
- given-names: Tyler
family-names: Hou
affiliation: UC Berkeley
- given-names: Alex
family-names: Rasmussen
email: [email protected]
affiliation: Bits on Disk
- given-names: Justin
family-names: Jaffray
email: [email protected]
affiliation: Sutter Hill Ventures
- given-names: David
family-names: Chu
affiliation: UC Berkeley
Expand All @@ -47,18 +44,11 @@ authors:
family-names: Power
orcid: 'https://orcid.org/0000-0002-0660-5110'
affiliation: UC Berkeley
- given-names: Amrita
family-names: Rajan
affiliation: UC Berkeley
- given-names: Rithvik
family-names: Panchapakesan
email: [email protected]
affiliation: UC Berkeley
repository-code: 'https://github.com/hydro-project/hydroflow/'
url: 'https://hydro.run/hydroflow'
url: 'https://hydro.run/'
abstract: >-
Hydroflow is the lattice-based dataflow runtime for the
Hydro language stack for programming distributed systems
Hydro is a language stack for programming distributed systems,
buil on top of DFIR, a lattice-aware dataflow system.
keywords:
- dataflow
- distributed systems
Expand Down
46 changes: 23 additions & 23 deletions CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,28 +1,28 @@
# Contributing to Hydroflow
# Contributing to Hydro

Thanks for your interest in contributing to Hydroflow! This is an experimental, research-driven
Thanks for your interest in contributing to Hydro! This is an experimental, research-driven
project which can make getting started a bit tricky. This guide will explain the project structure,
code style, commit messages, testing setups, and more to help you get started.

## Repository Structure

The Hydroflow repo is set up as a monorepo and [Cargo workspace](https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html).
The Hydro repo is set up as a monorepo and [Cargo workspace](https://doc.rust-lang.org/book/ch14-03-cargo-workspaces.html).
Relative to the repository root:

* `hydroflow` is the main Hydroflow package, containing the Hydroflow runtime. It re-exports the
surface syntax macros in `dfir_macro` and `dfir_lang`. The runtime is the "scheduled
layer" while the surface syntax compiler is the "compiled layer".
* `hydro_lang` and related (hydro_*) packages contain Hydro, which is a functional syntax built on
top of `hydroflow`.
* `dfir_datalog` provides a datalog compiler, based on top of the Hydroflow surface syntax.
* `dfir_rs` is the main DFIR package, containing the DFIR runtime. It re-exports the
flow syntax macros in `dfir_macro` and `dfir_lang`. The runtime is the "scheduled
layer" while the flow syntax compiler is the "compiled layer".
* `hydro_lang` and related (`hydro_*`) packages contain Hydro, which is a functional syntax built on
top of `DFIR`.
* `dfir_datalog` provides a datalog compiler, based on top of the DFIR flow syntax.
* `docs` is the [Hydro.run](https://hydro.run/) website. `website_playground` contains the
playground portion of the website, used for compiling Hydroflow in-browser via WASM.
* `benches` contains some microbenchmarks for Hydroflow and other frameworks.
* `design_docs` contains old point-in-time design docs for Hydroflow's architecture.
playground portion of the website, used for compiling DFIR in-browser via WASM.
* `benches` contains some microbenchmarks for DFIR and other frameworks.
* `design_docs` contains old point-in-time design docs for DFIR's architecture.

There are several subpackages included that are used by Hydroflow but are more general-purpose:
There are several subpackages included that are used by Hydro but are more general-purpose:

* `stageleft` is a framework for staged programming in Rust, used by `hydroflow_plus`.
* `stageleft` is a framework for staged programming in Rust, used by `hydro_lang`.
* `lattices` is a abstract algebra library, originally for lattice types.
* `variadics` is a crate for emulating variadic generics using tuple lists.
* `pusherator` is a rudimentary library providing push-based iterators.
Expand All @@ -37,13 +37,13 @@ There are auxillary repositories as well:

## Rust

Hydroflow should build on latest stable releases of Rust. However we develop on a pinned nightly
version, bumped up every month or two. The version is in `rust-toolchain.toml` which is
Hydro should build on latest stable releases of Rust. However we develop on a pinned nightly
version, which is updated every month or two. The version is in `rust-toolchain.toml` which is
automatically detected by `cargo`, so no special setup should be needed.

## Python

Some parts of the Hydroflow repo require a relatively recent version of Python 3, maybe 3.10 or
Some parts of the Hydro repo require a relatively recent version of Python 3, maybe 3.10 or
later. On Mac, installing directly from python.org may work if `brew install` doesn't.

### `wasm-bindgen`
Expand All @@ -70,7 +70,7 @@ git checkout -b $BRANCH_NAME `feature/$FEATURE_NAME`
.. make changes ..
git add ... # Add all changes
git commit # Commit changes
git push origin HEAD
git push origin HEAD
```

### Commit Messages
Expand All @@ -90,13 +90,13 @@ failing-- you should double-check that the snapshot diff matches what you expect

## Snapshot Testing

Hydroflow uses two types of snapshot testing: [`insta`](https://insta.rs/) and [`trybuild`](https://github.com/dtolnay/trybuild).
Insta provides general snapshot testing in Rust, and we mainly use it to test the Hydroflow graphs
generated from the surface syntax. These snapshots are of the [Mermaid](https://mermaid.js.org/) or
Hydro uses two types of snapshot testing: [`insta`](https://insta.rs/) and [`trybuild`](https://github.com/dtolnay/trybuild).
Insta provides general snapshot testing in Rust, and we mainly use it to test the DFIR graphs
generated from the flow syntax. These snapshots are of the [Mermaid](https://mermaid.js.org/) or
[DOT](https://graphviz.org/) graph visualizations rather than the graph datastructures themselves;
see `hydroflow/tests/snapshots`. The snapshots can be useful not just to track changes but also as
see `dfir_rs/tests/snapshots`. The snapshots can be useful not just to track changes but also as
a quick reference to view the visualizations (i.e. by pasting into [mermaid.live](https://mermaid.live/)).
`trybuild` is used to test the error messages in Hydroflow's surface syntax; see `hydroflow/tests/compile-fail`.
`trybuild` is used to test the error messages in DFIR's flow syntax; see `dfir_rs/tests/compile-fail`.

`insta` provides a CLI, `cargo insta` to run tests and review changes:
```shell
Expand Down
2 changes: 1 addition & 1 deletion LICENSE
Original file line number Diff line number Diff line change
Expand Up @@ -187,7 +187,7 @@
same "printed page" as the copyright notice for easier
identification within third-party archives.

Copyright 2022 Hydroflow Contributors
Copyright 2025 Hydro Contributors

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
Expand Down
14 changes: 7 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,12 @@
<a href="https://docs.rs/hydroflow/"><img src="https://img.shields.io/badge/docs.rs-Hydroflow-blue?style=flat-square&logo=read-the-docs&logoColor=white" alt="Docs.rs"></a>
</p>

Hydro is a novel distributed programming library for standard Rust. Hydro allows developers to build distributed systems that are efficient, scalable, and correct.
Hydro is a novel distributed programming library for standard Rust. Hydro allows developers to build distributed systems that are efficient, scalable, and correct.

Hydro integrates naturally into standard Rust constructs and IDEs, providing types and programming constructs for ensuring distributed correctness. Under the covers it provides a metaprogrammed compiler that optimizes for cross-node issues of scaling and data movement while leveraging Rust and LLVM for per-node performance.
Hydro integrates naturally into standard Rust constructs and IDEs, providing types and programming constructs for ensuring distributed correctness. Under the covers it provides a metaprogrammed compiler that optimizes for cross-node issues of scaling and data movement while leveraging Rust and LLVM for per-node performance.

We often describe Hydro via a metaphor: *LLVM for the cloud*. Like LLVM, Hydro is a layered compilation framework with a low-level Internal Representation language. In contrast to LLVM, Hydro focuses on distributed aspects of modern software.

We often describe Hydro via a metaphor: *LLVM for the cloud*. Like LLVM, Hydro is a layered compilation framework with a low-level Internal Representation language. In contrast to LLVM, Hydro focuses on distributed aspects of modern software.

<div align="center">
<img src="docs/static/img/hydro-stack.png" alt="Image description" width="200">
</div>
Expand Down Expand Up @@ -43,14 +43,14 @@ What's wanted, we believe, is a proper language stack addressing distributed con
- **Distributed Typechecking**: The type system of the language should enforce distributed correctness in the compiler, in ways that are visible in an IDE at time of authoring. For example, the types of data items should include their abstract locations, so that two items materialized at different nodes cannot be referenced together without an intervening construct for (async) communication.
- **Distributed Optimizations**: The compiler should be able to correctly optimize (transform) programs for distributed concerns: removing bottlenecks by flexibly assigning compute and data to different processes or clusters, while preserving program semantics.
- **Modularity**: The standard modularity of traditional programming—e.g. function calling abstractions—should work for distributed logic. For example, it should be possible to wrap a common cross-node construct like "heartbeats and timeouts", and invoke it as simply as one invokes a sequential function.
- **Native Testing Tools** for correctness. In today's standard practice, formal specification languages for testing (e.g. [TLA+](https://en.wikipedia.org/wiki/TLA%2B)) are separate from languages of implementation. We believe it should be possible to perform many kinds of formal testing (e.g. model checking) on the same code that is used in deployment.
- **Native Testing Tools** for correctness. In today's standard practice, formal specification languages for testing (e.g. [TLA+](https://en.wikipedia.org/wiki/TLA%2B)) are separate from languages of implementation. We believe it should be possible to perform many kinds of formal testing (e.g. model checking) on the same code that is used in deployment.

[Hydro](https://hydro.run) is a Rust library for distributed programming that is designed to address these goals.
[Hydro](https://hydro.run) is a Rust framework for distributed programming that is designed to address these goals.

# Learning More
The Hydro project's main website is at [https://hydro.run](https://hydro.run).

- **Docs**: There are docs for the [high-level Hydro language](https://hydro.run/docs/hydro/) and the low-level dataflow IR, [DFIR](https://hydro.run/docs/dfir), as well as the [HydroDeploy](https://hydro.run/docs/deploy) framework for launching Hydroflow+ programs.
- **Docs**: There are docs for the [high-level Hydro language](https://hydro.run/docs/hydro/) and the low-level dataflow IR, [DFIR](https://hydro.run/docs/dfir), as well as the [Hydro Deploy](https://hydro.run/docs/deploy) framework for launching Hydro programs.

- **Research Papers**: Our [research publications](https://hydro.run/research) are available on the project website. Some notable selections:
- The original Hydro vision paper from CIDR 2021: [New Directions in Cloud Programming](https://hydro.run/papers/new-directions.pdf)
Expand Down
2 changes: 1 addition & 1 deletion RELEASING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Releasing Guide

This is a guide on how to create releases for Hydroflow and all the other crates in this workspace.
This is a guide on how to create releases for all Hydro crates in this workspace.

We use the [`cargo-smart-release` crate](https://github.com/Byron/cargo-smart-release) for our
release workflow. Originally, cargo-smart-release [was part of gitoxide](https://github.com/Byron/gitoxide/pull/998)
Expand Down
2 changes: 1 addition & 1 deletion dfir_datalog/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ version = "0.10.0"
edition = "2021"
license = "Apache-2.0"
documentation = "https://docs.rs/dfir_datalog/"
description = "Datalog proc-macro for Hydroflow"
description = "Datalog proc-macro for DFIR"

[lints]
workspace = true
Expand Down
2 changes: 1 addition & 1 deletion dfir_lang/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ version = "0.10.0"
edition = "2021"
license = "Apache-2.0"
documentation = "https://docs.rs/dfir_lang/"
description = "Hydro's Data Flow Intermediate Representation (DFIR) implementation"
description = "Hydro's Dataflow Intermediate Representation (DFIR) implementation"

[lints]
workspace = true
Expand Down
6 changes: 3 additions & 3 deletions dfir_lang/src/graph/ops/demux.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,16 +14,16 @@ use crate::pretty_span::PrettySpan;

// TODO(mingwei): Preprocess rustdoc links in mdbook or in the `operator_docgen` macro.
/// > Arguments: A Rust closure, the first argument is a received item and the
/// > second argument is a variadic [`var_args!` tuple list](https://hydro-project.github.io/hydroflow/doc/hydroflow/macro.var_args.html)
/// > second argument is a variadic [`var_args!` tuple list](https://hydro.run/rustdoc/dfir_rs/macro.var_args.html)
/// > where each item name is an output port.
///
/// Takes the input stream and allows the user to determine which items to
/// deliver to any number of output streams.
///
/// > Note: Downstream operators may need explicit type annotations.
///
/// > Note: The [`Pusherator`](https://hydro-project.github.io/hydroflow/doc/pusherator/trait.Pusherator.html)
/// > trait is automatically imported to enable the [`.give(...)` method](https://hydro-project.github.io/hydroflow/doc/pusherator/trait.Pusherator.html#tymethod.give).
/// > Note: The [`Pusherator`](https://hydro.run/rustdoc/pusherator/trait.Pusherator.html)
/// > trait is automatically imported to enable the [`.give(...)` method](https://hydro.run/rustdoc/pusherator/trait.Pusherator.html#tymethod.give).
///
/// > Note: The closure has access to the [`context` object](surface_flows.mdx#the-context-object).
///
Expand Down
2 changes: 1 addition & 1 deletion dfir_lang/src/graph/ops/dest_sink.rs
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,7 @@ use super::{
///
/// If we only want to write a stream of bytes without delineation we can use the [`BytesCodec`](https://docs.rs/tokio-util/latest/tokio_util/codec/struct.BytesCodec.html).
///
/// In this example we use a [`duplex`](crate::tokio::io::duplex) as our `AsyncWrite` with a
/// In this example we use a [`duplex`](https://docs.rs/tokio/latest/tokio/io/fn.duplex.html) as our `AsyncWrite` with a
/// `BytesCodec`.
///
/// ```rustbook
Expand Down
4 changes: 2 additions & 2 deletions dfir_lang/src/graph/ops/mod.rs
Original file line number Diff line number Diff line change
Expand Up @@ -340,10 +340,10 @@ pub struct WriteContextArgs<'a> {
/// `dfir` crate name for `use #root::something`.
pub root: &'a TokenStream,
/// `context` ident, the name of the provided
/// [`dfir_rs::scheduled::Context`](https://hydro-project.github.io/hydroflow/doc/hydroflow/scheduled/context/struct.Context.html).
/// [`dfir_rs::scheduled::Context`](https://hydro.run/rustdoc/dfir_rs/scheduled/context/struct.Context.html).
pub context: &'a Ident,
/// `df` ident, the name of the
/// [`dfir_rs::scheduled::graph::Dfir`](https://hydro-project.github.io/hydroflow/doc/hydroflow/scheduled/graph/struct.Hydroflow.html)
/// [`dfir_rs::scheduled::graph::Dfir`](https://hydro.run/rustdoc/dfir_rs/scheduled/graph/struct.Dfir.html)
/// instance.
pub hydroflow: &'a Ident,
/// Subgraph ID in which this operator is contained.
Expand Down
2 changes: 1 addition & 1 deletion dfir_lang/src/graph/ops/source_interval.rs
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ use super::{
/// be scheduled as soon as possible.
///
/// Note that this requires the dfir instance be run within a [Tokio `Runtime`](https://docs.rs/tokio/1/tokio/runtime/struct.Runtime.html).
/// The easiest way to do this is with a [`#[dfir_rs::main]`](https://hydro-project.github.io/hydroflow/doc/hydroflow/macro.hydroflow_main.html)
/// The easiest way to do this is with a [`#[dfir_rs::main]`](https://hydro.run/rustdoc/dfir_rs/attr.main)
/// annotation on `async fn main() { ... }` as in the example below.
///
/// ```rustbook
Expand Down
6 changes: 3 additions & 3 deletions dfir_macro/build.rs
Original file line number Diff line number Diff line change
Expand Up @@ -228,8 +228,8 @@ const PREFIX: &str = "\
sidebar_position: 4
---
# Hydroflow's Operators
# DFIR Operators
In our previous examples we made use of some of Hydroflow's operators.
Here we document each operator in more detail. Most of these operators
In our previous examples we made use of some of DFIR's operators.
Here we document each operator in more detail. Many of these operators
are based on the Rust equivalents for iterators; see the [Rust documentation](https://doc.rust-lang.org/std/iter/trait.Iterator.html).";
Loading

0 comments on commit 6ab6252

Please sign in to comment.