Make TypedReference useful

[MichalStrehovsky]

Rationale

TypedReference is currently a second class citizen because using it equates to using undocumented C# keywords. We could make it more useful by adding methods to it that let normal code work with it.

We would be adding APIs that replace (and enhance) undocumented C# keyword __refvalue and __makref.

Proposed API

namespace System
{
    public struct TypedReference
    {
        public ref T AsRef<T>()
        {
            // Essentially "return ref __refvalue(this, T);" if C# allowed that.
        }
    }
}

Stretch goal:

namespace System
{
    public struct TypedReference
    {
        // This won't compile with C# because C# disallows returning a TypedReference
        // We need a Roslyn feature to make TypedReference returnable like other byref-like types
        public static TypedReference FromRef<T>(ref T value) { }
    }
}

[jkotas]

Related to https://github.com/dotnet/corefx/issues/14021

cc @atsushikan

[ghost]

public ref T AsRef(TypedReference reference)

I assume this is meant to be a nullary method.

[MichalStrehovsky]

I assume this is meant to be a nullary method.

Yes, fixed. Thanks for spotting!

[joshfree]

@steveharter can you look into this enhancement with https://github.com/dotnet/corefx/issues/29736 post-JSON?

[sakno]

From my point of view, this type can be used as dynamically strongly typed reference to the location in managed memory. It is somewhat trackable version of void* where you need to have managed pointer without knowledge of its actual type at compile time. unsigned int8& type can do the same but unclear to the developer because it tells you that you able to deference it into byte which is wrong at least for write operation. From other side, typedref allows to control type safety for the managed pointer even if underlying type is unknown. This is what you want to achieve by GetRawData and GetRawArrayData methods where the type of the reference is unknown.

According with this concept it would be great to see refactored and modernized version of this type with the following characteristics:

1. Allow to create typedref using static factory method without magic of undocumented __makeref keyword
2. Allow to return typedref from the method
3. Allow to pass typedref by reference
4. Allow to declare typedref as a member of another by-ref-like value
5. Rewrite internals of typedref in the way when this value type can be treated like regular by-ref-like value type
6. Allow to obtain raw managed pointer (probably, protected with ref readonly) to the stored reference.
7. Allow to obtain reference to the field without knowing actual type of the field

At first, its two IntPtr fields can be replaced by

private readonly RuntimeTypeHandle _type;
private readonly ByReference<byte> _reference;

Now we have regular by-ref like value type without special handling from CLR side.

At second, we need new methods to make this type useful:

public static TypedReference Make<T>(ref T value);

Allows to created typed reference from the managed pointer. Information about type T stored as _type field. It provides alternative to undocumented __makeref keyword in C#.

public static TypedReference Make<T>(in T obj, RuntimeFieldHandle field);
public static TypedReference Make(TypedReference obj, RuntimeFieldHandle field);

These methods allow to obtain typed reference to the specified field. This is what proposed by @MichalStrehovsky about GetRawData. You don't need to compute offset to the field used low-level magic and API is not low-level as mentioned by @jkotas. And still we have dynamically typed access to the field. in T is used instead of object to avoid boxing if T is value type; or you can use overloaded alternative if T is not known at compile time.

public bool IsTypeOf<T>();

Intrinsic method that allows to check whether the typed reference is of type T.

public ref T AsRef<T>();

As proposed by Michal previously. However, this method can be dangerous because may refer to initonly field. In this case, ref readonly return type modifier can be used. If developer wants to mutate the value by reference then he can use Unsafe.AsRef<T>(in T obj);

public ref readonly byte RawData { get; }

Allows to obtain raw managed pointer to the referenced value. Can be useful for low-level operations such as pointer arithmetic or situations where underlying type of the referenced value irrelevant. ref readonly here to avoid dangerous writes.

public int TypeSize { get; }

Returns the size of the underlying type, in bytes. The behavior is equal to sizeof IL instruction.

public TypedReference Upcast<T>() where T : class;
public TypedReference Upcast(Type type);

Allows to change the underlying type to subclass. It has the following behavior:

• If underlying type is value type then return empty reference or throw InvalidCastException
• If underlying type is a base class of the target type then return reference to the same memory location but different underlying type
• If underlying type is reference type that is convertible to the target type then return empty reference or throw InvalidCastException.

public bool IsEmpty { get; }

If typed reference is empty (initialized as default).

public void CopyTo(TypedReference destination);
public bool TryCopyTo(TypedReference destination);

Copies the value stored at the reference to the memory referenced by another TypedReference. Types of both references should be equal.

Some of these methods can be implemented in pure IL without special treatment from CLR side.

Unsafe class can provide additional methods to work with typed reference:

public static TypedReference AddOffset(TypedReference obj, int offset);

Returns typed reference to the specified offset in the memory. Can be used for fast access to the array elements without knowledge about its element type.

public static TypedReference Unbox(object obj);

Returns typed reference to the boxed value; or empty, if obj is not a boxed value type.

Of course, all these additions stay backward compatible and existing methods should not be removed.

There is one drawback: at this moment it is possible to declare unmanaged pointer to typedref, see 25697. My proposal breaks this ability.

[GSPP]

@sakno I'd be curious to hear what use cases you have in mind? I do not dispute that this type might be useful. But I fail to imagine concrete use cases.

[sakno]

@sakno I'd be curious to hear what use cases you have in mind? I do not dispute that this type might be useful. But I fail to imagine concrete use cases.

At least GetRawData and GetRawArrayData. Generally, proposed methods can be used as fast alternative to Reflection-based member access:

1. Reading and writing field values without knowledge about its type at compile-time
2. Boxing-free way to invoke method, see this comment.

As I pointed in my previous post, typedref is a way to represent a dynamically strongly typed reference to the location in managed memory. void*allows to represent arbitrary location in the memory but the pointer cannot be tracked by GC. void& type doesn't exist in CLR and can be fully replaced by typedref. byte& can do the same but it doesn't hold the information about runtime type and cannot be dereferenced safely.

JVM has similar concept of fast access to the field or array element using VarHandle

[steveharter]

Currently there is active design work in 6.0 around fast invoke, which may use TypedReference in the manner above (e.g. "AsRef", "FromRef") provided the current compiler restrictions around passing it can be removed (causing CS1601 or CS1955).

@MichalStrehovsky are you aware of any reasons why the Roslyn compiler needs to continue to prevent TypedReference from being passed to other methods (either byref or byval). I see you changed it to be a ref struct a year ago which I hope addresses potential misuse, but I'm not sure about any other intrinsic dependencies or potential GC issues.

Also I currently don't see any active issues around TypedReference in Roslyn, so this is an early attempt to gather information before pursuing.

[jkotas]

@steveharter TypedReference has the same problem as the internal ByReference<T> type. Both of them are structs with ref fields. The current C# lifetime rules for ref-structs are not able to model ref fields safely.

https://github.com/dotnet/csharplang/blob/master/proposals/low-level-struct-improvements.md talks about the problem and the proposed solution.