oracle proposal

[Tibo-lg]

This is a proposal for oracle specification.

[Tibo-lg]

Added some details about periodic events.

[Christewart]

Related to #44

[ariard]

Concept ACK, I think this lay out well each concept of the oracle subsystem : event, oracle, URI, templates, signatures.

@LLFourn I guess our confusion on Slack was you're speaking about the event namespace (the URI path) when I was thinking about the logical encoding of events in template (my relation between two quantities rough scheme). IMO, that's two different encoding, and the good thing with opt-in for TLV as a serialization format it would make easy to introduce new logical encoding, beyond the current ones (enumeration, scoped range, exponent-mantissa range)

[ariard]

Either unix epoch or block height ?

[Tibo-lg]

Specified that it has to be unix epoch. I personally don't think that it makes sense to use block height here but open for arguments.

[ariard]

Points to BOLT 1 Type-Length-Value Format ?

[Tibo-lg]

Done

[ariard]

ECDSA/Schnorr ? Over which curve(s) ?

[Tibo-lg]

Good question, not sure if it matters so much here since this is mainly for accountability, maybe @LLFourn has some argument in favor of one or another?

[LLFourn]

use the same signature scheme as the anticipated signature scheme i.e. bip340 schnorr

[Tibo-lg]

Specified that it should follow BIP 340

[ariard]

Uniqueness compared to what ? All other events issued by the same oracle ?

[Tibo-lg]

Yes, tried to specify more clearly.

[ariard]

Shouldn't this be called an event template, as events are the objects issued by the entity oracle ? A template is a definition of a data structure format, in this context an oracle isn't a data structure.

Also I think you can have an introduction section before (## Oracle templates) where you explain that oracles are entities issuing event outcomes. Events are digital mapping to real-word fact. Each event outcome announcement is committed through a template before its realization. Templates are resources leveraged by DLC participants to establish a set of DLCs. These resources are discovered through a namespace, presently categorized URI paths.

[Tibo-lg]

I agree about the naming, updated. Also added some introduction along the lines of what to mention, but it's maybe a bit repetitive with the beginning of the second section now. Let me know what you think.

[LLFourn]

Concept ACK, I think this lay out well each concept of the oracle subsystem : event, oracle, URI, templates, signatures.

@LLFourn I guess our confusion on Slack was you're speaking about the event namespace (the URI path) when I was thinking about the logical encoding of events in template (my relation between two quantities rough scheme). IMO, that's two different encoding, and the good thing with opt-in for TLV as a serialization format it would make easy to introduce new logical encoding, beyond the current ones (enumeration, scoped range, exponent-mantissa range)

FWIW I don't think there's confusion. I am precisely saying that we shouldn't separate these two things. I think from the URI for the event you can generate all outcomes and the client can fully determine the semantics of each outcome without making any HTTP requests.

I will provide a motivation for this when I make a counter proposal.

@Tibo-lg thanks this document is good.

[LLFourn]

Once the client has determined this iinfo how do you get the details of an event for a specific date/time? Do you append the time onto the url?

[Tibo-lg]

Yeah that's pretty much the idea :)

[LLFourn]

I assume this includes R values as well?

[Tibo-lg]

Yes!

[LLFourn]

To be clear: In this proposal if I do a GET to /sport/nba/s2019/w24/HOU_PHI/vs my client should be perfectly happy to get back a response that describes a range or an exponent/mantissa. i.e. they can infer nothing from the URL itself.

[Tibo-lg]

Yes that is correct.

[Tibo-lg]

Added a bit to try to make it clearer.

[LLFourn]

I expect that the public key is cached or even embedded in the app and not communicated with the R value (the public key is static and should only be fetched once).

I think "event announcement" would be a better name for this with the template being the name for the part of the announcement that describes the outcomes (this caused me confusion below). I'm not sure if "template" is right. Usually a template is something you fill in. Maybe just "outcome description" even though that's not great either.

[Tibo-lg]

I see the point about the public key, at first I was thinking similarly, but it is also really convenient that everything is self contained. Otherwise we also need to standardize a way to retrieve public keys (not that it's impossible though).

And I agree that template is probably not a good name, I think originally we used "template" to describe something that an oracle would fill in to describe an event, but given how I presented things it definitely doesn't fit. Also are you saying that it should be divided in two parts (announcement and description) or am I misunderstanding?

[Tibo-lg]

Went with "event descriptor" for now, still not super happy with it though.

[LLFourn]

Where does the information go about which location this is at and whose tomorrow it is? You don't just sign sunny right? You must sign something like "It was sunny in Paris on 2020-08-15" or am I missing something.

[Tibo-lg]

For me the "in paris on 2020-08-15" should be encoded in the URI. So you would retrieve the information from "/weather/france/paris/2020-08-15" for example and then the signature would indeed just be over "sunny". (And I realize that the URI should probably be part of what the oracle commit to prior to the event).

[Tibo-lg]

Specified that the URI should be included in the attestation signature.

[LLFourn]

Wait so you decided to put it in the attestation after all? i.e. not just sign "sunny". I think that would be better otherwise in order for a system to know the outcome of an event they would not be able to tell just from the signature and the message.

[Tibo-lg]

What I referred as attestation signature is not the signature over the event outcome but the commitment of the oracle to provide the event outcome in the future (sorry if I got the terms wrong)

[LLFourn]

The terms I am using are:

1. announcement: When the oracle announces its intention to attest to an event outcome (provides R values)
2. attestation: The decision on the outcome through revealing the corresponding signature scalar.

IMO It is sub-optimal to sign just "sunny" since, given the signature from the attestation you cannot tell what fact the oracle was attesting to just -- that the outcome is "sunny". You would have to combine this with the announcement signature to infer what the actual fact was. In my vision oracles are also useful as a source of historical facts rather than just DLCs so it would be nice to have everything in the one signature.

[Tibo-lg]

Thanks I'll try to follow this as well then, I like this terminology.

I don't have strong opinion about what is being signed so I'm ok with adding some extra data to it.

[nkohen]

What about further partition? If it is truly a large range the current scheme doesn't help much (compared to signing each digit in the base individually). Currently, if I'm understanding correctly, the mantissa could have quite a bit of precision necessary.

[Tibo-lg]

As mentioned here at the end I agree that decimal decomposition is better. But I don't really see a precision issue with mantissa approach, as I don't think that you have to manipulate it as a number, but just as a string (I don't think you need to do any computation on the result, and if you do, the issue seem to be the same regardless of the representation).

[nkohen]

I see, so if the base being used in the exponent is the same as the original representation then I agree there is no precision issue.

That said, if you agree I'd love to see the updated version of this that includes the more general decomposition

[Tibo-lg]

Updated to use decomposition

[nkohen]

I see, so if the base being used in the exponent is the same as the original representation then I agree there is no precision issue.

That said, if you agree I'd love to see the updated version of this that includes the more general decomposition

[nkohen]

I think this is not descriptive enough for most users.

[Tibo-lg]

Let me know if the new version is more understandable.

[nkohen]

This doesn't really help anyone who doesn't already know what you mean

[nkohen]

That is to say, a description of the actual thing being signed here is necessary

[Tibo-lg]

ditto

[devrandom]

It might be good to explicitly enumerate the CETs for this case. I think I see how to efficiently construct log(n) of them, given that each CET can represent a boolean AND of several R values. But I'm not sure that it will be obvious to many readers.

I assume the basic idea is to construct CETs as follows. (I use the notation Rn=m to mean that the outcome m is signed with the R value Rn)

• R0=1 - outcome is underflow
• R0=2 - outcome is underflow
  ...
• R0=5 && R1=1 && R2=0 - this is the desired range, but this needs to be further decomposed, assuming the distribution to each party is different based on the exact price event outcome
• R0=5 && R1=1 && R2>0 - 9 overflow cases
• R0=5 && R1>1 - 8 more overflow cases
• R0=6 - overflow
  ...
• R0=10 - overflow

[devrandom]

It might be good to represent both equality and open-ended range for the exponent, which allows for complete safety. i.e. could represent R0=5 and also R0_ge=5 (representing the fact that the exponent is greater or equal to 5 using a different R value). This allows a CET that deals with arbitrary overflows.

[Tibo-lg]

Thanks for the suggestion, I updated roughly following it.

It might be good to represent both equality and open-ended range for the exponent, which allows for complete safety. i.e. could represent R0=5 and also R0_ge=5 (representing the fact that the exponent is greater or equal to 5 using a different R value). This allows a CET that deals with arbitrary overflows.

That's a good suggestion, though one issue I can see with this is that it makes it difficult for an oracle to update an event as I describe in the example. So if at time t the oracle releases 5 R values, but at time t+1 because of large price increase he wants to now use 6 for newly created contract, the R0_ge=5 is not useful for these newly created contracts. Maybe it could be possible to at that time create yet another R0_ge to be used for when price goes above 6 digits, but from an implementation perspective it feels like becoming a bit complex, though you are right that this would provide complete safety which is nice.

It would of course also be possible for the oracle to just create a different event, but it also makes things more complicated IMHO.

[devrandom]

Ah, right, I should have called it R_ge_5, since the oracle can't safely sign more than one thing with a single R value.

Also, it may be that exponential increases in an event value are so unlikely that this is overkill.

[Tibo-lg]

It might be an overkill, but at the same time it's also a bit neater. I haven't fully made up my mind but I think it's really worth considering.

[devrandom]

Another potential approach is related to your idea about different events. Any particular event could have a "saturation point" - e.g. if the Oracle doesn't expect the price to go above 99999, then part of the event definition could be that it saturates at that point. i.e. the oracle signs the limit if the real-world event goes above it.

If things move strongly in that direction, then a new event could be created with a higher saturation point. But that doesn't mean the existing contracts missed out on anything, because they already had saturated outcomes in their CETs, most likely at an even lower value.

[Tibo-lg]

Yeah I'm starting to think this might be the direction to go, though instead of saturation point I would maybe still keep an extra R0 that can be just 0 or 1 (1 meaning it's overflowed). In that case I think we need to think about having possible versions for events, so that we can have a same "base ID" (which for example would represent BTC/USD at a certain date for a given exchange) but multiple versions in case of large price fluctuations. One thing we lose is the small optimization in the number of CETs but as @LLFourn noted it's quite small.

I will try to update tomorrow with this approach, let me know if you have other comments.

[Tibo-lg]

Finally got around to update this, @devrandom if you have time to have a look, I finally used the "sign the highest possible number" approach as you suggested (I think it's the easiest and also cleanest).

[devrandom]

👍

[ariard]

ACK c19a5e2 pending on this, just to be sure we don't have issue.

[Christewart]

I think we should write out examples for both range and large range, this doesn't have to be done in this PR, but if it isn't done in this PR I want to file an issue. Please give a 👍 if you want me to file an issue, otherwise I will assume you will put it in this PR.

Nvm this is done and i am just blind 🙄

EDIT2: It does look like we need a large range event example with a signed value that contains negative numbers https://github.com/discreetlogcontracts/dlcspecs/blob/c19a5e2f7dd459dac77da4ec4d649cf9f51a1682/Oracle.md#example-btcusd-rate

[LLFourn]

I implemented something that was inspired by this over the last couple of days.
The differences are:

• I use "json" to encode the oracle_event for easier development and iteration
• The nonces are not in the event_descriptor, so it is more of an outcome_descriptor. I felt the descriptor should only do the job of telling you what the possible outcomes are and how many signatures will be used to attest to the outcome. The nonces are communicated in a sibling field. This allows the descriptor to be reused even for signature schemes that don't have nonces e.g. BLS signatures.
• renamed event_maturity_epoch to expected_outcome_time which I think is a better name.
• [edit] also no public keys in the signed data

Here's what my oracle's api looks like for a simple binary outcome:

curl -s h00.ooo/random/2020-10-26T17:56:00/heads_tails?left-win |jq

{
  "announcement": {
    "oracle_event": {
      "encoding": "json",
      "data": "{\"id\":\"/random/2020-10-26T17:56:00/heads_tails?left-win\",\"expected_outcome_time\":\"2020-10-26T17:56:00\",\"descriptor\":{\"type\":\"enum\",\"outcomes\":[\"heads_win\",\"tails_win\"]},\"nonces\":[\"0abc60c1fa4329f9340766f302211c5732564e009d9068608f8b04f4cce69786\"]}"
    },
    "signature": "c37e1600d9848802fbc917d73fcf7f701e11916de1df42f6f377fa4628b9fd12a3f135bc978f5f0236db0a5c9f89c73191d6aec2827d61ad27d3b757d93d5e9f"
  },
  "attestation": {
    "outcome": "heads_win",
    "scalars": [
      "bf156ff1c44672349d2273bcc782d93be69e55bfcb31e6169afc03d147906e83"
    ],
    "time": "2020-10-26T17:56:00"
  }
}

The oracle_event by itself:
curl -s h00.ooo/random/2020-10-26T17:56:00/heads_tails?left-win | jq -r .announcement.oracle_event.data | jq

{
  "id": "/random/2020-10-26T17:56:00/heads_tails?left-win",
  "expected_outcome_time": "2020-10-26T17:56:00",
  "descriptor": {
    "type": "enum",
    "outcomes": [
      "heads_win",
      "tails_win"
    ]
  },
  "nonces": [
    "0abc60c1fa4329f9340766f302211c5732564e009d9068608f8b04f4cce69786"
  ]
}

Note this is just informational and not a demand to change anything.

[Tibo-lg]

@LLFourn

The nonces are not in the event_descriptor, so it is more of an outcome_descriptor. I felt the descriptor should only do the job of telling you what the possible outcomes are and how many signatures will be used to attest to the outcome. The nonces are communicated in a sibling field. This allows the descriptor to be reused even for signature schemes that don't have nonces e.g. BLS signatures.

Actually I think originally the nonces were not in the event descriptor but in the oracle event. IIRC I moved it to the oracle event because of the TLV serialization since some event types can have multiple while others have only one, and it felt error prone to not enforce single nonce for event that require it. But based on what you say, maybe we should reconsider this and move them to the oracle event structure? @benthecarman what do you think?

renamed event_maturity_epoch to expected_outcome_time which I think is a better name.

My opinion on this is very weak, so I'll let @benthecarman decide since I think he chose the original name.

[benthecarman]

But based on what you say, maybe we should reconsider this and move them to the oracle event structure? @benthecarman what do you think?

I think both are valid. I originally had the nonces outside the event descriptor for the same reason that @LLFourn describes. However, I do like having them in the descriptor so the number is enforced at the type level. Since I think they are both equally valid maybe we should change to what @LLFourn is doing so we are more compatible?

renamed event_maturity_epoch to expected_outcome_time which I think is a better name.

I think event_maturity_epoch makes more sense, maturity is meant to signify that it is the earliest an event can happen, so for example if you're signing for "Do aliens invade in 2020", you can sign immediately if they do, rather than needing to wait until the end of the year. Also, epoch is meant to make it more explicit we are using epoch time.

[benthecarman]

I think both are valid. I originally had the nonces outside the event descriptor for the same reason that @LLFourn describes. However, I do like having them in the descriptor so the number is enforced at the type level. Since I think they are both equally valid maybe we should change to what @LLFourn is doing so we are more compatible?

Thinking about this more, we could just make 2 OracleEvent TLVs, one for multi nonce and one for single nonce. That could help keep things more error prone

[LLFourn]

I think event_maturity_epoch makes more sense, maturity is meant to signify that it is the earliest an event can happen, so for example if you're signing for "Do aliens invade in 2020", you can sign immediately if they do, rather than needing to wait until the end of the year. Also, epoch is meant to make it more explicit we are using epoch time.

It sounds like we are talking about different things. For expected_outcome_time I was trying to communicate when the event was expected (not the earliest possible). Why is the earliest useful to clients?

However, I do like having them in the descriptor so the number is enforced at the type level. Since I think they are both equally valid maybe we should change to what @LLFourn is doing so we are more compatible?

I agree that the type safety it gives you is good. You can still enforce this at the type level just use pack the nonces after the descriptor without a type and length, then use the number of nonces implied by the descriptor to decode. I think it's really useful not to have the nonces in there since then you can: (i) re-use the descriptors across different curves and signature schemes (ii) check that two descriptors describe the same outcomes through a simple equality check.

[Tibo-lg]

Put the nonces in the oracle_event.

Thinking about this more, we could just make 2 OracleEvent TLVs, one for multi nonce and one for single nonce. That could help keep things more error prone

I think that even though I agree it could be nicer, it would make things a bit too complicated. And I think we should be able to catch any inconsistency at deserialization time, and have internal type handle the type safety.

[LLFourn]

This needs nb_digits now.

Each descriptor must state how many nonces are required for it. e.g. this would be nb_digits plus one of it's signed, the others would be 1.
Then when decoding nb_nonces must be equal to this number otherwise fail.

Also what about renaming this to digit_decomposition_descriptor? I am concerned that someone later figures out a better way of doing a "large range" than digit decomposition.

[Tibo-lg]

Good point, added nb_digits, and also updated the description as I had not removed the nonce part from the text.

Updated to digit_decomposition_descriptor, I'm fine with this name, I agree it's a bit more explicit.

[nkohen]

Just wording nits here in case you want to make a couple wording changes before we merge!

[LLFourn]

I consider the hashing to be part of the signing algorithm. I think we just need to describe in another document a sign(keypair, tag, message) function which tags the message (e.g. tagged hash) and uses BIP340 sign internally. This document can then just say what the message and tag should be without mentioning tagged at all. Currently it's missing a tag e.g. oracle/outcome. Of course this doesn't exist yet so consider it just a comment.

I also think the message should not just be the raw outcome string (as discussed before) but should include the event id somehow. The format I am using is {event_id}.{i}={outcome_str} where i is the index (i.e. digit) of the signature. I don't really mind if we specify this later as it's an easy thing to change.

[Tibo-lg]

I updated too mention that the encoded value is passed to "a signing algorithm". I haven't understood if you want the message format ({event_id}.{i}={outcome_str}) to be mentioned here or somewhere else?

[LLFourn]

It can probably be here, just in a future revision.

[nkohen]

I opened an issue so that we can take care of this in a future PR

[LLFourn]

It doesn't make sense have the oracle public key here. It creates the hazard of someone using the public key here to validate the signature. They must already know the oracle's public key and must not decode oracle_event until they have validated the signature. It would be good to stress this in the document.

[Tibo-lg]

Where would you put it then? The reason for having it in this message is for when you get the data from a third party (not directly from the oracle). In that case you need to have some id for the oracle, and the public key seems the best to me. Am I misunderstanding what you mean?

[LLFourn]

Applications can decide where to put it. In some applications they may decide that oracles are identified by their public key and send that key along with the announcement. In other application they may decide oracles are identified by something else. In other cases their may be clients that are bound only to one oracle so they don't need an identifier. Regardless of whether the public key is sent, there's no need for the oracle to put this in the announcement data.

Does this make sense? If you guys are intent on using the public key as an oracle identifier in the p2p protocol then perhaps just slot the public key right next to this data in https://github.com/discreetlogcontracts/dlcspecs/blob/master/Messaging.md

[nkohen]

I think for now we can just move this pubkey to the announcement TLV?

[benthecarman]

If we are removing the pub key from the tlv we should add a recommendation somewhere that the event id should have some reference to the oracle (whether it be the pubkey, a website, or just something like @OutcomeObserver).

[nkohen]

I think it is fair to say that an event TLV is worthless without the signature, and that the public key should come with the signature (in the announcement for example). I'm not sure what you mean by remove from TLV as we would add it to the announcement?

[Tibo-lg]

Moved the public key to the oracle announcement as @nkohen suggested.

[LLFourn]

I noted to @Tibo-lg that I don't think the oracle_announcement struct really belongs here as it looks more like a p2p struct that clients send to each other. An oracle would not communicate its public key with each event. This can be fixed later.

[ariard]

Post-merge ACK 5380516

[ariard]

IMO, you could insert a footnote, event to a GH discussion to enlighten future document readers. Otherwise, this kind of sentence is a bit useless as you don't know what "complex constructions where considered".