flesh out the tutorial

doc/tutorials/adnvanced-ensemble.md

@@ -1,5 +1,7 @@
 # Advanced Actor Ensembles
 
+TODO words
+
 ## Ensemble Domains
 
 TODO words
@@ -13,11 +15,21 @@ TODO words
 TODO words
 
 ### Using a Standalone httpd
+Here we walk through an example www project and how to serve it with `gerbil httpd server`.
+
+First, let's compile our site's code -- see [src/tutorial/advanced-ensemble](https://github.com/mighty-gerbils/gerbil/tree/master/src/tutorial/advanced-ensemble) for the code.
 ```
 1$ pushd site/project
 1$ gerbil build
 1$ popd
+```
+
+Now let's configure the httpd. Specifically:
+- we set `www` as the root directory for the site
+- we enable servlets
+- we set a handler for the code we just build
 
+```
 1$ pushd site
 1$ gerbil httpd -G project/.gerbil config --root www --enable-servlets --handlers '(("/handler" . :demo/handler))'
 
@@ -32,7 +44,10 @@ handlers:
 (("/handler" . :demo/handler))
 enable-servlets:
 #t
+```
 
+And now let's start it and interact with it.
+```
 1$ gerbil httpd -G project/.gerbil server
 ...
 
@@ -43,6 +58,21 @@ enable-servlets:
 
 ### Using an httpd ensemble
 
+A single server is just that: one process, with one thread of control
+serving requests.  If we want to take advantage of multicore machines
+with process isolation, we can spawn more and multiplex on the socket
+using `SO_REUSEPORT` (the server does it by default).
+
+We can do this very easily by constructing an ensemble, and by doing
+that we take advantage of supervision and the all the available
+tooling from `gxensemble`.
+
+Here is how we can configure an httpd ensemble:
+- we configure the ensemble root directory
+- we configure the ensemble domain and worker domain
+- we configure the number of workers
+- and just pass the through the httpd config.
+
 ```
 1$ pushd site
 1$ gerbil httpd -G project/.gerbil config --ensemble --ensemble-root root --ensemble-domain /test --worker-domain /test/www -n 2 --root www --enable-servlets --handlers '(("/handler" . :demo/handler))'
@@ -79,10 +109,18 @@ domain:
 /test
 root:
 "root"
+```
 
+This configuration will preload and supervise, restarting as needed, 2 workers.
+The all we have to do is:
+```
 1$ gerbil httpd -G project/.gerbil ensemble
 ...
+```
 
+So let's interact with our ensemble, and notice the multiplexing by
+the server identifier in the response:
+```
 2$ ps auxw | grep gerbil
 2$ curl http://localhost:8080/
 2$ curl http://localhost:8080/servlets/hello.ss
@@ -91,10 +129,24 @@ root:
 ```
 
 ## Local Ensembles
-TODO words
 
+What about services composed of things other than httpds? Can't we run
+a general supervised ensemble that is open ended and can dynamically
+add or remove servers?
+
+The answer is a resounding yes; in fact `gerbil httpd ensemble` runs a
+generic supervisor and which just happens to be configured for httpd.
+In the folling we see how we can run an httpd ensemble with a standalone supervisor.
+
+First, let's configure the ensemble:
+- first we configure the httpd as usual -- notice the use of `-G env/local` so that
+  the httpd configuration is placed in the standard service configuration path.
 ```
 1$ gerbil httpd -G env/local config  --root www --enable-servlets --handlers '(("/handler" . :demo/handler))'
+```
+
+And then we configure the ensemble to know about the `httpd` role:
+```
 1$ gerbil ensemble -G env/local config ensemble -D '/test' --root root
 1$ gerbil ensemble -G env/local config role --role httpd --exe gerbil --prefix '("httpd" "server")' --policy restart --env www --application httpd
 
@@ -128,14 +180,26 @@ roles:
                  (("/handler" . :demo/handler))
                  enable-servlets:
                  #t)))))
+```
 
-
+And that's it, now we can start a supervisor and spawn httpd server process with it.
+```
 1$ gerbil ensemble -G env/local supervisor
 ...
 
+```
+
+In order to easily interact with the ensemble supervisor, let's also configure the
+environment to know about our supervisor:
+```
 2$ gerbil ensemble -G env/local env supervisor '(supervisor . /test)'
 2$ gerbil ensemble -G env/local env known-servers --add  '(supervisor . /test)'
+```
+
+And now we can interact with the supervisor directly:
+```
 2$ gerbil ensemble -G env/local control list-servers --pretty
+(((registry . /test) 2348146 running))
 
 2$ gerbil ensemble -G env/local control get-ensemble-config --pretty
 (config:
@@ -210,8 +274,15 @@ roles:
                   (("/handler" . :demo/handler))
                   enable-servlets:
                   #t))))))
+```
 
+As we can see, the only server within the ensemble under the
+supervisor is the automatically spawned registry, which facilitates
+local interactions between servers in the ensemble.
 
+Before we can spawn our httpds, we also need to _upload_ the necessary code
+and content:
+```
 2$ cd site/project/.gerbil/
 2$ tar czvf ../../env-www.tar.gz lib
 
@@ -220,38 +291,79 @@ roles:
 
 2$ gerbil ensemble -G env/local control upload --fs site/fs-www.tar.gz ""
 2$ gerbil ensemble -G env/local control upload --env site/env-www.tar.gz www
+```
+
+And that's it, now we are ready to spawn our httpd servers as workers:
+```
 2$ gerbil ensemble -G env/local control start-workers -d /test/www httpd httpd 2
 2$ gerbil ensemble -G env/local control list-servers --pretty
+(((httpd-0 . /test/www) 2348366 running)
+ ((httpd-1 . /test/www) 2348367 running)
+ ((registry . /test) 2348146 running))
+```
 
+And we can interact with them:
+```
 2$ ps auxw | grep gerbil
 2$ curl http://localhost:8080/
 2$ curl http://localhost:8080/servlets/hello.ss
 2$ curl http://localhost:8080/handler
+```
 
+Finally, we can shutdown the entire ensemble, including the supervisor:
+```
 2$ gerbil ensemble -G env/local control shutdown
 ```
 
 ## Distributed Ensembles
-TODO words
 
+Naturally, we are not limited to controlling a local ensemble behind a
+supervisor.  We can just as easy spawn a distributed ensemble that
+runs in multiple hosts, all running their own supervisor.
+
+The individual supervisors themselves can be spawned and supervised as
+a `systemd` service.
+
+In this tutorial, we build an ensemble with 3 hosts: 2 hosts serving
+http requests with a (local) httpd ensemble each, and another host
+acting as a load balancer using the example `rlb` program.
+
+See [src/tutorial/advanced-ensemble/rlb](https://github.com/mighty-gerbils/gerbil/tree/master/src/tutorial/advanced-ensemble/rlb) for the `rlb` code.
+
+::: tip Note
+The IP addresses shown here were 3 linode servers, configured just for demonstration
+purposes and no longer exist.
+Please set up your own servers when following the tutorial!
+:::
+
+First, a bit of configuration: if we are going to manage a distributed ensemble we need to use TLS. So let's generate a CA and create certificates for our supervisors:
 ```
 1$ gerbil ensemble -G env/private ca setup --domain demo.ensemble.internal
 1$ gerbil ensemble -G env/private ca cert '(supervisor . /demo/linode1)'
 1$ gerbil ensemble -G env/private ca cert '(supervisor . /demo/linode2)'
 1$ gerbil ensemble -G env/private ca cert '(supervisor . /demo/linode3)'
+```
 
+Next, let's configure the supervisors for each host: notice that we
+enable public access by passing the address where we listen with TLS:
+```
 1$ gerbil ensemble -G env/linode1 config ensemble -D /demo/linode1 --root root --public 0.0.0.0:4999
 1$ gerbil ensemble -G env/linode2 config ensemble -D /demo/linode2 --root root --public 0.0.0.0:4999
 1$ gerbil ensemble -G env/linode3 config ensemble -D /demo/linode3 --root root --public 0.0.0.0:4999
-
+```
+Next, let's package and upload the necessary environment for our supervisors:
+```
 1$ gerbil ensemble -G env/private package -o env/linode1.tar.gz -C env/linode1/ensemble/config '(supervisor . /demo/linode1)'
 1$ gerbil ensemble -G env/private package -o env/linode2.tar.gz -C env/linode2/ensemble/config '(supervisor . /demo/linode2)'
 1$ gerbil ensemble -G env/private package -o env/linode3.tar.gz -C env/linode3/ensemble/config '(supervisor . /demo/linode3)'
 
 1$ scp env/linode1.tar.gz [email protected]:
 1$ scp env/linode2.tar.gz [email protected]:
 1$ scp env/linode3.tar.gz [email protected]:
+```
 
+Next, let's unpack the environment and run our supervisors:
+```
 # linode1
 2$ ssh [email protected]
 2$ mkdir env
@@ -269,11 +381,16 @@ TODO words
 # linode3
 4$ ssh [email protected]
 4$ ...
+```
 
+Again, in order to facilitate the interaction, we setup our private environment:
+```
 1$ gerbil ensemble -G env/private env known-servers --set '(supervisor . /demo/linode1)' '(tls: "172.233.56.134:4999")'
 1$ gerbil ensemble -G env/private env known-servers --set '(supervisor . /demo/linode2)' '(tls: "172.233.56.175:4999")'
 1$ gerbil ensemble -G env/private env known-servers --set '(supervisor . /demo/linode3)' '(tls: "172.233.56.211:4999")'
-
+```
+Next, we configure and start our httpd servers:
+```
 1$ gerbil httpd -G env/tmp config  --root www --enable-servlets --handlers '(("/handler" . :demo/handler))'
 1$ gerbil ensemble -G env/tmp/linode1 config role --role httpd --exe gerbil --prefix '("httpd" "server")' --policy restart --env www --application httpd -C env/tmp/config/httpd
 
@@ -385,7 +502,10 @@ $ gerbil ensemble -G env/private control get-ensemble-config -S '(supervisor . /
 1$ curl http://172.233.56.175:8080/
 1$ curl http://172.233.56.175:8080/handler
 1$ curl http://172.233.56.175:8080/servlets/hello.ss
+```
 
+And finally, let's configure and start the load balancer:
+```
 1$ cd rlb
 1$ gerbil build
 
@@ -400,11 +520,29 @@ proxies: ("172.233.56.134:8080" "172.233.56.175:8080")
 1$ gerbil ensemble -G env/private control upload -S '(supervisor . /demo/linode3)' --exe rlb/.gerbil/bin/rlb rlb
 1$ gerbil ensemble -G env/private control update-ensemble-config -S '(supervisor . /demo/linode3)' env/tmp/linode3/ensemble/config
 1$ gerbil ensemble -G env/private control start-workers -S '(supervisor . /demo/linode3)' -d www rlb rlb 2
+```
 
+Try it out:
+```
 1$ curl http://172.233.56.211:8080/
 1$ curl http://172.233.56.211:8080/servlets/hello.ss
+```
 
+And something for fun, let's ping an rlb server behind a supervisor and then start a repl on it, even though it doesn't itself has a public TLS address:
+```
 1$ gerbil ensemble -G env/private ping -s -S '(supervisor . /demo/linode3)' '(rlb-0 . /demo/linode3/www)'
 
 1$ gerbil ensemble -G env/private repl -s -S '(supervisor . /demo/linode3)' '(rlb-0 . /demo/linode3/www)'
 ```
+
+## Ensemble Orchestration
+
+TODO orchiestrator and meta-supervision
+
+This functionality is planned for v0.19
+
+## More Ensemble Services
+
+TODO vault, broadcast, resolver
+
+This functionality is planned for v0.19