You'll want to include these namespaces in your C# code for the following examples to work.
using Kusto.Language;
using Kusto.Language.Symbols;
using Kusto.Language.Syntax;
You can parse a Kusto query or control command by using the KustoCode.Parse method.
// parse only
var query = "T | project a = a + b | where a > 10.0";
var code = KustoCode.Parse(query);This produces a KustoCode instance that contains the parsed syntax tree.
The tree can be navigated using a variety of API's
like GetDescendants, GetAncestors, GetChild, Parent, WalkNodes, GetTokenAt, or GetNodeAt.
If you wanted to find all the places where a particular name was referenced in the syntax tree
you can search the tree for all places where a NameReference node has that particular name.
// search syntax for a name reference of "a"
var referencesToA = code.Syntax.GetDescendants<NameReference>(n => n.SimpleName == "a");
// there are two NameReference nodes with name "a" and one NameDeclaration
Assert.AreEqual(2, referencesToA.Count);However, notice that the query contains two different columns named a.
The column a supposedly defined by table T and the column a declared by the project operator.
Given only syntax, it is not possible to distinguish that the column a referred to in the where operator
is not the same as the column declared by the table.
In order to correctly distinguish between the two, you can request that the parser perform semantic analysis and identify for you which column is which.
Semantic analysis is the process that determines what exactly all the names refer to and checks the query for semantic errors.
When semantic analysis has been performed you will be able to determine which piece of syntax refers to which exact column, variable, function or table.
To parse with semantic analysis enabled, use the KustoCode.ParseAndAnalyze method.
This method allows you to specify a GlobalState instance that contains the definition for the database tables
and functions that the query may reference. These definitions are called symbols.
var globals = GlobalState.Default.WithDatabase(
new DatabaseSymbol("db",
new TableSymbol("T", "(a: real, b: real)")));
var query = "T | project a = a + b | where a > 10.0";
var code = KustoCode.ParseAndAnalyze(query, globals);Now when you navigate the syntax tree you can access the ReferencedSymbol and ResultType properties
that tell you what is being referenced and the type of any expression.
For this example, you can simply check the ReferencedSymbol property to see if it exactly matches the
instance of the ColumnSymbol that was defined as part of table T when you declared the schema.
// search syntax tree for references to specific columns
var columnA = globals.Database.Tables.First(t => t.Name == "db").GetColumn("a");
var referencesToA = code.Syntax.GetDescendants<NameReference>(n => n.ReferencedSymbol == columnA);
// there is only one reference to the column named "a" from the table "T"
Assert.AreEqual(1, referencesToA.Count);
If you wanted to see if any errors were detected during parsing or semantic analysis,
you can call the GetDiagnostics method.
var diagnostics = code.GetDiagnostics();
if (diagnostics.Count > 0) { ... }The method GetDiagnostics will return all the syntactic and semantic errors found in the query.
It is possible to get diagnostics for a query that has not undergone semantic analysis.
In this case, you would only get the syntax errors found during parsing.
Not all diagnostics are errors. Check the Severity property to see if it is an error, warning or other type of diagnostic.
You can use the parsed query to discover all the columns explicitly referenced by the query that originate from a database table.
public static HashSet<ColumnSymbol> GetDatabaseTableColumns(KustoCode code)
{
var columns = new HashSet<ColumnSymbol>();
SyntaxElement.WalkNodes(code.Syntax,
n =>
{
if (n.ReferencedSymbol is ColumnSymbol c
&& code.Globals.GetTable(c) != null)
{
columns.Add(c);
}
});
return columns;
}This function uses the SyntaxElement.WalkNodes method to traverse all the nodes in the syntax tree in lexical order, top down.
It checks each node to see if it references a ColumnSymbol and then uses the GetTable method on the GlobalState
object to check if the column is a member of a known database table.
The following examples shows using the GetDatabaseTableColumns function on a simple query.
var globals = GlobalState.Default.WithDatabase(
new DatabaseSymbol("db",
new TableSymbol("Shapes", "(id: string, width: real, height: real)"),
new FunctionSymbol("TallShapes", "{ Shapes | where width < height; }")
));
var query = "TallShapes | where width > 5 | project id, width";
var code = KustoCode.ParseAndAnalyze(query, globals);
var dbColumns = GetDatabaseTableColumns(code);The result dbColumns contains the column symbols for the width and id columns, because these
are the only columns explicitly referenced within the body of the query.
However, since the query is actually over the function TallShapes instead of the Shapes table directly,
this naive approach to finding column references does not include any additional columns referenced by the called function.
You can improve on this approach with a somewhat more elaborate function that recursively analyzes the called function bodies too.
public static HashSet<ColumnSymbol> GetDatabaseTableColumns(KustoCode code)
{
var columns = new HashSet<ColumnSymbol>();
GatherColumns(code.Syntax);
return columns;
void GatherColumns(SyntaxNode root)
{
SyntaxElement.WalkNodes(root,
fnBefore: n =>
{
if (n.ReferencedSymbol is ColumnSymbol c
&& code.Globals.GetTable(c) != null)
{
columns.Add(c);
}
else if (n.GetCalledFunctionBody() is SyntaxNode body)
{
GatherColumns(body);
}
},
fnDescend: n =>
// skip descending into function declarations since their bodies will be examined by the code above
!(n is FunctionDeclaration)
);
}
}This advanced function uses the GetCalledFunctionBody method found on syntax nodes.
For nodes that refer to user or database functions (like a function call node),
it will return the root node of the function body.
You can analyze this separate function body for additional column references
by recursively calling the GatherColumns method.
The function also supplies the fnDescend delegate to the SyntaxElemenet.WalkNodes method,
using it to skip over the bodies of any let statement declared functions inside the query.
This will avoid duplicating work now handled by the recursion.
Now when the function is used to find all the columns referenced by the query,
it will include the column height referenced by the TallShapes function itself.
In a manner similar to finding all the columns referenced in a query you can find all the database tables too.
public static HashSet<TableSymbol> GetDatabaseTables(KustoCode code)
{
var tables = new HashSet<TableSymbol>();
SyntaxElement.WalkNodes(code.Syntax,
n =>
{
if (n.ReferencedSymbol is TableSymbol t
&& code.Globals.IsDatabaseTable(t))
{
tables.Add(t);
}
else if (n is Expression e
&& e.ResultType is TableSymbol ts
&& code.Globals.IsDatabaseTable(ts))
{
tables.Add(ts);
}
});
return tables;
}This simple approach checks for all tables explicitly named in the query and any table that might be indirectly introduced into the query via a function call by checking the result type of expressions.
However, like with columns, this approach is naive and will not discover that
the TallShapes function references the Shapes table on your behalf.
You can improve upon it by using the same recursive technique used to find the columns.
public static HashSet<TableSymbol> GetDatabaseTables(KustoCode code)
{
var tables = new HashSet<TableSymbol>();
GatherTables(code.Syntax);
return tables;
void GatherTables(SyntaxNode root)
{
SyntaxElement.WalkNodes(root,
fnBefore: n =>
{
if (n.ReferencedSymbol is TableSymbol t
&& code.Globals.IsDatabaseTable(t))
{
tables.Add(t);
}
else if (n is Expression e
&& e.ResultType is TableSymbol ts
&& code.Globals.IsDatabaseTable(ts))
{
tables.Add(ts);
}
else if (n.GetCalledFunctionBody() is SyntaxNode body)
{
GatherTables(body);
}
},
fnDescend: n =>
// skip descending into function declarations since their bodies will be examined by the code above
!(n is FunctionDeclaration)
);
}
}
If a column is part of a database table, you can discover that table using the GetTable method
on the GlobalState that includes the database table definition.
var table = code.Globals.GetTable(column);
If a table symbol is declared as part of a database, you can discover which database is associated with the table
using the GetDatabase method on the GlobalState that includes the definition of the database and table.
var database = code.Globals.GetDatabase(table);
You can discover the cluster that a database belongs to using the GetCluster method on the GlobalState
that includes the definition of both the cluster and database symbols.
var cluster = code.Globals.GetCluster(database);In order to have the parser understand the existence of any particular database tables or functions, it must be told about them first.
You tell the parser about the tables and functions by adding DatabaseSymbol instances to the GlobalState instance you use with the ParseAndAnalyze method.
You can declare tables by constructing TableSymbol instances.
var shapes = new TableSymbol("Shapes", "(id: string, width: real, height: real)");You can declare functions by constructing FunctionSymbol instances.
Functions can be declared with our without parameters.
var tallshapes = new FunctionSymbol("TallShapes", "{ Shapes | width < height; }");
var shortshapes = new FunctionSymbol("ShortShapes", "(maxHeight: real)", "{ Shapes | height < maxHeight; }");Once you have all the tables and function symbols you can create a DatabaseSymbol.
var mydb = new DatabaseSymbol("mydb", shapes, tallshapes, shortshapes);In order to use your database symbol it must be in the GlobalState instance.
Since GlobalState is an immutable type, you add to it by making new instances using the With methods.
One way to get a GlobalState to know about your database is to modify the default database symbol that is in scope when you query is analyzed.
var globalsWithMyDb = GlobalState.Default.WithDatabase(mydb);Another way is to add a cluster definition. It can contain multiple databases.
var mycluster = new ClusterSymbol("mycluster.kusto.windows.net", mydb);Once you have the cluster, you can either add it as the default cluster and the database as the default database or add it as a separate cluster (not the default).
var globalsWithMyDefaultCluster = GlobalState.Globals.WithCluster(mycluster).WithDatabase(mydb);
var globalsWithMyClusterAdded = GlobalState.Globals.AddOrReplaceCluster(mycluster);If you add the cluster that is not he default, you must use the cluster() function to access it in the query.
Likewise, if you have a database that is not the default, you must use the database() function to access it.
Declaring all these symbols manually is a bit tedious and unnecessary if you already have an existing Kusto database defined. It is possible to construct all the necessary symbols by using Kusto itself to query the database and request all the schema information.
Unfortunately, there are no API's provided in the library to do this for you.
There is, however, another library available on nuget that includes the necessary API's to load the symbols: https://www.nuget.org/packages/Kusto.Toolkit/
The source code to this library is available at: https://github.com/mattwar/Kushy
You can give the parser additional functions and aggregates to know about even though they might not really exist in the server
by adding them to the global state instance you use when you call the ParseAndAnalyze method.
var fnFake = new FunctionSymbol("fake", ScalarTypes.Real, new Parameter("x", ScalarTypes.Long), new Parameter("y", ScalarTypes.Long));
var globals = GlobalState.Default.WithFunctions(globals.Functions.Concat(new [] {fnFake}).ToArray());
var code = KustoCode.ParseAndAnalyze("print fake(10)", globals); var fnMinMax = new FunctionSymbol("minmax", ScalarTypes.Real, new Parameter("x", ScalarTypes.Real));
var globals = GlobalState.Default.WithAggregates(globals.Aggregates.Concat(new [] {fnMinMax}).ToArray());
var code = KustoCode.ParseAndAnalyze("T | summarize minmax(c)", globals);Likewise, if you remove functions or aggregates from the global state's list the parser will produce an error when they are used.