Meta Programming in ANKHOR
Meta Programming is a concept that dates back to languages such as LISP or Smalltalk. The term is used for programs that write or modify other programs – or perform operations on themselves. Some languages have a dedicated “meta” level outside the language, such as the C++ template system other such as LISP make no real distinction between programs and data.
It is an extremely powerful concept and most modern programming systems have at least some elements of meta programming in their toolbox.
Graphs and Operators as Values
Operators and Graphs of Operators can be used as first class data type values in ANKHOR. They can be created, inspected, placed in other data structures, separated, combined and executed. The simplest way to create and use an operator as a value is to copy a graph as value using the context menu.p>It can then be pasted into another graph and executed using the lambda operator.
These operator values can also be pasted into tables or tag lists for reference by name or other criteria. The opposite way of copying an operator value as operator can also be performed using the context menu.
One example of such a usage can be found in the implementation of the “Compress Rows” operator of the data cube library. The table is used to select the appropriate operator for the aggregation of rows with the same keys.
These operator values are similar to function-pointers in traditional programming languages and can be used for the same purposes.
A second common technique to create operator values is the application of the compile operator. It receives a string with ANKHOR functional language source code (ANKL) and generates an operator value for execution.
This technique allows any kind of source to be compiled into operators and is most useful when combined with a user input for the source. One can also use the expression operator to generate a constant operator by using the FUNC keyword.
This is frequently used to generate tag lists of operators for method tables:
The three compiling operators “compile”, “expression” and “function” provide an easy way to compile e.g. mathematical formulas that would be awkwardly implemented using a hand built operator graph.
Recursion and This Graph
Recursion in a flow sheet is implemented using dynamically generated operator values. This is accomplished using the “this” operator. It generates an operator value that is identical to the macro it is directly embedded in. This value is then passed into the next recursion level and then executed using the lambda operator. The classic example for recursion is the factorial calculation:
The “this” operator can also be used to create a non recursive operator value that is passed out of the macro it is embedded in:
The problem here is the additional “this” output in the generated operator value, but this can easily be removed with an operator from the “CompiledExpressions” library
This operator does not only remove the additional output, it removes the “this” operator as well and creates a simplified vanilla version of the macro.
This “this” removing operator is implemented using meta-operators.
Meta operators are operators that use operator values as input or create operator values themselves. The two simple ones are used to split an operator or a macro into its components:
Both operators return the type, name and connectors of the operator. The split macro operator returns the list of child operators and internal connections as well. There is also a set of meta-operators to build new operators that have a matching set of inputs:
This example generates a dynamic “adder” operator that adds a constant to its input value. This is not yet a very useful operator but shows the general principle of building larger operators from smaller ones.
Many library operators perform more complex operations on operator values using a combination of the split and build meta operations. One example is the “cxstripthis” operator of the previous example another one would be “cxpartialeval” for partial evaluation. This operator receives an operator value and a tag list with the input values that are constant and know upfront. It creates a new operator that has these input values connected to constants.
Partial evaluation is not only useful to create operators with a reduced number of inputs, it can also create faster and simpler operators with the help of the “optimize” operator:
The built in optimize-macro operator performs an extensive set of optimizations such as constant-folding, loop-unrolling, macro-inlining or strength reduction. The usual process with constructed operators is to first build the complete complex operator and then optimize only once just before execution.
Operator Values and Remote Execution
One important use case of operator values is remote execution. The operator value that is generated in this scenario is not executed locally but on the ANKHOR remote execution server (ARES). ARES contains a complete ANKHOR execution engine and can be run on server grade hardware in the data centre or on a remote system. Operators that provide ARES functionality are not directly working with the data, but instead create objects that include an expression tree. This tree is converted into an operator and sent to ARES when the user inspects one of the results or invokes the final query operator.
A simple sample graph might look like this
When we open the OSQLQuery operator, we notice that it invokes an operator that is stored in the private section of the query object:
This is another use of operator values, implementing dynamic linking for objects. The query operator takes the tag based expression tree and converts it into an operator that is then executed on the ARES system.
This is the parse tree. One can easily deduce the original graph from its structure:
And here is the generated and optimized operator graph:
All macros are expanded and reduced to the most basic built in operators. This graph is then sent to the ARES server for execution.
ANKHOR is a completely reflective meta programming system, in which the ANKHOR graphs can be create, inspected and manipulated by the same “language”. While most ANKHOR users will not walk into meta programming land further than compiled operators, it provides a very powerful tool for library authors and advanced ANKHOR users.
Download FlowSheet and sample data (ZIP archive)