Storm Reference - Pivoting

Pivot operations are performed on the output of a previous Storm operation such as a lift or filter. Pivot operators are used to perform pivot operations by navigating from one set of nodes to another based on a specified relationship. Most often, this relationship is two properties (primary and / or secondary) that share the same value, and that may also share the same Type.

That is, other than some specialized use cases (such as pivoting to or from tags), most pivots involve navigating between the following kinds of properties:

  • primary to secondary;

  • secondary to primary;

  • secondary to secondary; or

  • primary to primary.

Note

Primary to primary property pivots are a specialized use case that is commonly handled using Raw Pivot Syntax, below.

Where the source and target properties have the same value and the same type, Storm can leverage Synapse’s Type Awareness to simplify pivot operations and identify implicit relationships. For example, type awareness is used for Implicit Pivot Syntax and also for “wildcard” pivot operations (specialized use cases for the Pivot Out Operator and the Pivot In Operator).

The pivot operations available within Storm are:

Note

Light edges represent a special use case within the Synapse hypergraph; navigating (traversing) light edges (Lightweight (Light) Edge) is included here as a “pivot-like” operation.

See Storm Reference - Document Syntax Conventions for an explanation of the syntax format used below.

See Storm Reference - Type-Specific Storm Behavior for details on special syntax or handling for specific data types.

Pivot Out Operator

The pivot out operator ( -> ) is the primary Storm pivot operator. The pivot out operator is used for:

  • primary to secondary property pivots,

  • secondary to primary property pivots,

  • secondary to secondary property pivots, and

  • “wildcard” pivot out - pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any nodes, leveraging Synapse’s Type Awareness.

Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes are covered separately below.

Syntax:

  • Primary to Secondary:

    <query> -> <form> : <prop>

  • Secondary to Primary:

    <query> : <prop> -> <form>

  • Secondary to Secondary:

    <query> : <prop> -> <form> : <prop>

  • “Wildcard” Pivot Out:

    <query> -> *

Examples:

Pivot from primary property (<form> = <valu>) to secondary property (<prop> = <pval>):

  • Pivot from a set of domains to all of their subdomains regardless of depth (i.e., from a domain to all of the domains where the inbound domain is a zone):

<inet:fqdn> -> inet:fqdn:zone

  • Pivot from a set of domains to the DNS A records for those domains:

<inet:fqdn> -> inet:dns:a:fqdn

Pivot from secondary property (<prop> = <pval>) to primary property (<form> = <valu>):

  • Pivot from a set of DNS A records to the resolution IP addresses contained in those records:

<inet:dns:a> :ipv4 -> inet:ipv4

Pivot from secondary property (<prop> = <pval>) to secondary property (<prop> = <pval>):

  • Pivot from the WHOIS records for a set of domains to the DNS A records for the same domains:

<inet:whois:rec> :fqdn -> inet:dns:a:fqdn

“Wildcard” pivot out - pivot from all secondary properties to the primary properties of the equivalent forms (<prop> = <pval> to <form> = <valu>):

  • Pivot from a set of WHOIS records to all nodes whose primary property equals any of the secondary properties of the WHOIS record:

<inet:whois:rec> -> *

In the example above, the pivot would navigate from the :fqdn, :registrar, and :registrant secondary properties of the inet:whois:rec nodes (for example) to the associated inet:fqdn, inet:whois:rar, and inet:whois:reg nodes.

Usage Notes:

  • When pivoting from a secondary property (<prop> = <pval>), the secondary property must be specified using the relative property name only (:baz vs. foo:bar:baz). If you specify the full property name before the pivot, Storm interprets that as an additional lift (i.e., <inet:dns:a> inet:dns:a:fqdn -> inet:fqdn would be interpreted as “take a set of DNS A records from an initial query, lift all DNS A records with an :fqdn property (i.e., every DNS A node in the Cortex), and then pivot to the associated FQDN nodes”).

  • Pivoting out using the asterisk wildcard ( * ) is sometimes called a refs out pivot because it pivots from all secondary properties of the inbound nodes to all nodes referenced by those properties. That is, for each inbound node, the “refs out” pivot will pivot from the node’s secondary properties to all the nodes that have a primary property equal to that type and value.

  • Pivoting using the wildcard is based on strong data typing within the Synapse data model and Synapse’s type awareness, so will only pivot out to properties that match both <type> and <valu> / <pval>. This means that the following nodes will not be returned by a wildcard pivot out:

    • Nodes with matching <valu> / <pval> but of different <type>. For example, if a node’s secondary property is a string (type <str>) that happens to contain a valid domain (type <inet:fqdn>), a wildcard pivot out from the node with the string value will not return the inet:fqdn node.

    • Digraph (edge) nodes, whose properties are of type <ndef> (node definition, or <form>,<valu> tuples). See Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes for details on pivoting to / through those forms.

  • It is possible to perform an explicit pivot between properties of different types. For example: <inet:dns:query> :name -> inet:fqdn

  • See Pivot Out and Walk for a more comprehensive alternative to the wildcard pivot out.

Pivot In Operator

The pivot in ( <-) operator is similar to but separate from the pivot out ( ->) operator. The pivot in operator pivots to the set of nodes that reference the current set of nodes.

Logically, any pivot in operation can be expressed as an equivalent pivot out operation. For example, the following two pivots would be functionally equivalent:

  • Pivot from a set of domains to their associated DNS A records:

    <inet:fqdn> -> inet:dns:a:fqdn

  • Use “pivot in” to navigate from a set of domains to the DNS A records that reference a set of domains:

    <inet:fqdn> <- inet:dns:a:fqdn

Because of this equivalence, and because “left to right” logic is generally more intuitive, only pivot out has been fully implemented in Storm. (The second example, above, will actually return an error.) The pivot in operator exists, but is only used for certain special case pivot operations:

Syntax:

  • “Wildcard” Pivot In

    <query> <- *

Example:

Pivot from all primary properties to all nodes with an equivalent secondary property (<form> = <valu> to <prop> = <pval>):

  • Pivot from a set of domains to all nodes with a secondary property that references the domains:

<inet:fqdn> <- *

In this example, the pivot might return nodes with secondary properties such as inet:whois:email:fqdn, inet:dns:ns:zone, inet:dns:query:name:fqdn, and so on.

Usage Notes:

  • Pivoting in using the asterisk wildcard ( * ) is sometimes called a refs in pivot because it pivots from the inbound nodes to all nodes that reference those nodes. That is, for each inbound node, the “refs in” pivot will pivot from the primary property of a node to all nodes that have a secondary property equal to that type and value.

  • Pivoting in using the wildcard will return an instance of a node for each matching secondary property. For example, where a node may have the same <pval> for two different secondary properties (such as :domain and :zone on an inet:fqdn node), the pivot in will return two copies of the node. Results can be de-duplicated using the Storm uniq command.

  • Pivoting using the wildcard is based on strong data typing within the Synapse data model and Synapse’s type awareness, so will only pivot in from properties that match both <type> and <valu> / <pval>. This means that the following nodes will not be returned by a wildcard pivot in:

    • Nodes with matching <valu> / <pval> but of different <type>. For example, if a node’s primary property (such as a domain, type <inet:fqdn>) - happens to be referenced as as a different type (such as a string, type <str>) as a secondary property of another node, a wildcard pivot in to the inet:fqdn node will not return the node with the string value.

    • Digraph (edge) nodes, whose properties are of type <ndef> (node definition, or <form>,<valu> tuples). See Pivot to Digraph (Edge) Nodes and Pivot Across Digraph (Edge) Nodes for details on pivoting to / through those forms.

  • Other than digraph (edge) node navigation / traversal, pivot in can only be used with the wildcard ( * ). That is, pivot in does not support specifying a particular target form:

    inet:fqdn=woot.com <- inet:dns:a:fqdn

    The above query will return an error. A filter operation (see Storm Reference - Filtering) can be used to downselect the results of a wildcard pivot in operation to a specific set of forms:

    inet:fqdn=woot.com <- * +inet:dns:a

  • See Pivot In and Walk for a more comprehensive alternative to the wildcard pivot in.

Pivot With Join

The pivot and join operator ( -+> ) performs the specified pivot operation but joins the results with the inbound set of nodes. That is, the inbound nodes are retained and combined with the results of the pivot.

Another way to look at the difference between a pivot and a join is that a pivot operation consumes nodes (the inbound set is discarded and only nodes resulting from the pivot operation are returned) but a pivot and join does not consume the inbound nodes.

The pivot and join operator is used to retain the inbound nodes in any of the following cases:

  • primary to secondary property pivots,

  • secondary to primary property pivots,

  • secondary to secondary property pivots, and

  • “wildcard” pivot out - pivot from any / all secondary properties of the inbound set of nodes to the equivalent primary property of any nodes.

Syntax:

  • Primary to Secondary

    <query> -+> <form> : <prop>

  • Secondary to Primary

    <query> : <prop> -+> <form>

  • Secondary to Secondary

    <query> : <prop> -+> <form> : <prop>

  • “Wildcard” Pivot Out and Join

    <query> -+> *

Examples:

Pivot and join from primary property (<form> = <valu>) to secondary property (<prop> = <pval>):

  • Return a set of domains and all of their immediate subdomains:

<inet:fqdn> -+> inet:fqdn:domain

Pivot and join from secondary property (<prop> = <pval>) to primary property (<form> = <valu>):

  • Return a set of DNS A records and their associated IP addresses:

<inet:dns:a> :ipv4 -+> inet:ipv4

Pivot and join from secondary property (<prop> = <pval>) to secondary property (<prop> = <pval>):

  • Return the WHOIS records for a set of domains and the DNS A records for the same domains:

<inet:whois:rec> :fqdn -+> inet:dns:a:fqdn

“Wildcard” pivot out and join - pivot from all secondary properties to the primary properties of the equivalent forms (<prop> = <pval> to <form> = <valu>):

  • Return a set of WHOIS records and all nodes whose primary property equals any of the secondary properties of the WHOIS record:

<inet:whois:rec> -+> *

Usage Notes:

  • A pivot out and join operation follows the same caveats and constraints as the standard Pivot Out Operator.

Traverse (Walk) Light Edges

The traverse (walk) light edges operator ( -(<verb>)> or <(<verb>)- ) is used to traverse from a set of inbound nodes to the set of nodes they are linked to by the specified light edge(s). Because a light edge is not a node, the navigation is technically a “traversal” of the light edge as opposed to a property-to-property pivot.

Similar to an edge in a traditional directed graph, light edges have a “direction” (i.e., the relationship represented by a light edge is “one way”). From a Storm syntax perspective, light edges can be traversed in either direction.

Syntax:

  • Walk - Single Light Edge

    <query> -( <verb> )> * | <form>

    <query> <( <verb> )- * | <form>

  • Walk - Multiple Light Edges

    <query> -( ( <verb1> , <verb2> [ , <verb3> …] ) )> * | <form>

    <query> <( ( <verb1> , <verb2> [ , <verb3> …] ) )- * | <form>

  • Walk - Any Light Edge (Wildcard)

    <query> -( * )> * | <form>

    <query> <( * )- * | <form>

Examples:

Traverse the “refs” light edge from an article to the FQDNs “referenced” by the article:

<media:news> -(refs)> inet:fqdn

Traverse the “refs” light edge from an article to all of the nodes “referenced” by the article:

<media:news> -(refs)> *

Traverse the “hasip” light edge from an IPv4 address to the CIDR block(s) the IP is part of:

<inet:ipv4> <(hasip)- inet:cidr4

Traverse the “hasip” and “ipwhois” light edges from an IPv4 address to any nodes linked via those light edges (i.e., typically the CIDR block(s) the IP is part of and the netblock registration record(s) for the IP):

<inet:ipv4> <((hasip, ipwhois))- *

Traverse any / all light edges from an article to all nodes linked by any light edge:

<media:news> -(*)> *

Usage Notes:

  • The traversal syntax allows specification of a single verb, a list of verbs, or the “wildcard” / asterisk ( * ) to reference any / all light edge verbs that may be present.

  • There are no light edges (i.e., specific light edge verbs) defined in a Cortex by default. Users can create and define their own according to their needs.

  • The Storm model, edges, and lift.byverb commands can be used to work with light edges in a Cortex.

Pivot Out and Walk

The pivot out and walk (traverse) light edges operator ( --> * ) combines a wildcard pivot out (“refs out”) operation ( -> * ) with a wildcard walk light edges operation ( -(*)> ).

Syntax:

<query> —-> *

Examples:

Pivot from an IP netblock registration record to all nodes referenced by the record’s secondary properties and all nodes linked to the record by light edges:

<inet:whois:iprec> --> *

Usage Notes:

  • The pivot out and walk operator can only be used with a wildcard ( * ); it is not possible to specify a particular form as the target of the operation. A filter operation can be used to refine the results of the pivot and walk operation if necessary.

  • The pivot and walk operators (pivot out and walk / pivot in and walk) are useful for “exploring” data in a Cortex as they will return all the nodes “next to” the working set of nodes (subject to Type Awareness) without requiring the user to have specific knowledge of the data model.

  • The Storm tee command can be used to perform concurrent pivot in and walk / pivot out and walk operations on an inbound set of nodes:

<query> | tee { --> * } { <-- * }

Pivot In and Walk

The pivot in and walk (traverse) light edges operator ( <-- * ) combines a wildcard pivot in (“refs in”) operation ( <- * ) with a wildcard walk light edges operation ( <(*)- ).

Syntax:

<query> <—- *

Examples:

Pivot from a set of IP addresses to all nodes that reference the IPs and all nodes linked to the IPs by light edges:

<inet:ipv4> <-- *

Usage Notes:

  • The pivot in and walk operator can only be used with a wildcard ( * ); it is not possible to specify a particular form as the target of the operation. A filter operation can be used to refine the results of the pivot and walk operation if necessary.

  • The pivot and walk operators (pivot out and walk / pivot in and walk) are useful for “exploring” data in a Cortex as they will return all the nodes “next to” the working set of nodes (subject to Type Awareness) without requiring the user to have specific knowledge of the data model.

  • The Storm tee command can be used to perform concurrent pivot in and walk / pivot out and walk operations on an inbound set of nodes:

<query> | tee { --> * } { <-- * }

Pivot to Digraph (Edge) Nodes

Digraph (edge) nodes (Edge (Digraph) Form) are of type edge or timeedge. These nodes (forms) are unique in that their primary property value is a pair of node definitions (type Ndef) - that is, <form>, <valu> tuples. (timeedge forms are comprised of two <form>, <valu> tuples and an additional <time> value). Each <form>, <valu> tuple from the primary property is broken out as secondary property :n1 or :n2 of type <ndef>. This means that pivoting to and from digraph nodes is a bit different than pivoting to and from nodes whose properties are a simple <valu> or <pval>.

Note

Edge nodes are not formally deprecated, but the use of light edges (see Lightweight (Light) Edge) is now preferred over edge nodes.

Syntax:

<query> -> <edge> | <timeedge> [:n2]

<query> -+> <edge> | <timeedge> [:n2]

<query> <- <edge> | <timeedge>

Examples:

Pivot out from a set of nodes whose ndefs (<form>, <valu>) are the first element (:n1) in a set of a digraph nodes:

  • Pivot out from a person node to the set of digraph nodes representing things that person “has”:

<ps:person> -> edge:has

  • Pivot out from a person node to the set of timeedge digraph nodes representing places that person has been to (and when):

<ps:person> -> edge:wentto

Pivot in from a set of nodes whose ndefs (<form>, <valu>) are the second element (:n2) in a set of a digraph nodes:

  • Pivot in from an article to the set of digraph nodes representing things that “have” the article (e.g., people or organizations who authored the article):

<media:news> <- edge:has

Usage Notes:

  • To simplify working with digraph nodes and their ndef properties, Storm makes some assumptions (optimizations) when using the pivot out and pivot in operators:

    • When pivoting to or from a set of nodes to a set of digraph nodes, pivot using the ndef (<form>,<valu>) of the inbound nodes and not their primary property (<valu>) alone.

    • When pivoting out to a digraph node, the inbound nodes’ <form>,<valu> ndef will be the first element (:n1) of the digraph. You must explicitly specify :n2 as the target property to pivot using the second element.

    • When pivoting in to a digraph node, the inbound nodes’ <form>,<valu> ndef will be the second element (:n2) of the digraph. It is not possible to pivot into the :n1 value.

  • Pivoting to / from digraph nodes is one of the specialized use cases for the pivot in ( <-) operator, however the primary use case of pivot in with digraph nodes is reverse edge traversal (see Pivot Across Digraph (Edge) Nodes). See Pivot In Operator for general limitations of the pivot in operator.

Pivot Across Digraph (Edge) Nodes

Because digraph nodes represent generic edge relationships, analytically we are often more interested in the nodes on “either side” of the edge than in the digraph node itself. For this reason, the pivot operators have been optimized to allow a syntax for easily navigating “across” these digraphs (edges).

Note

Edge nodes are not formally deprecated, but the use of light edges (see Lightweight (Light) Edge) is now preferred over edge nodes.

Syntax:

<query> -> <edge> | <timeedge> -> * | <form>

<query> <- <edge> | <timeedge> <- * | <form>

Examples:

  • Traverse a set of edge:has nodes to pivot from a person to all the things the person “has”:

<ps:person> -> edge:has -> *

  • Traverse a set of edge:wentto nodes to pivot from a person to the locations the person has visited:

<ps:person> -> edge:wentto -> *

Usage Notes:

  • Storm makes the following assumptions to optimize the two pivots:

    • For pivots out, the first pivot is to the digraph nodes’ :n1 property and the second pivot is from the digraph nodes’ :n2 property.

    • For pivots in, the first pivot is to the digraph nodes’ :n2 property and the second pivot is from the digraph nodes’ :n1 property.

  • Pivoting “across” the digraph nodes still performs two pivot operations (i.e., to the digraph nodes and then from them). As such it is still possible to apply an optional filter to the digraph nodes themselves before the second pivot.

Pivot to Tags

Pivot to tags syntax allows you to pivot from a set of nodes with tags to the set of syn:tag nodes representing those tags. This includes:

  • pivot to all leaf tag nodes,

  • pivot to all tag nodes,

  • pivot to all tag nodes matching a specified pattern, and

  • pivot to tag nodes matching an exact tag.

See the Analytical Model document for additional discussion of tags as nodes (syn:tag nodes) and tags as labels applied to other nodes.

Syntax:

<query> -> # [ * | <tag> .* | <tag> ]

Examples:

Pivot to all leaf tag nodes:

  • Pivot from a set of domains to the syn:tag nodes for all leaf tags applied to those domains (i.e., the longest / final tag in each tree applied to each node):

<inet:fqdn> -> #

Pivot to ALL tag nodes:

  • Pivot from a set of files to the syn:tag nodes for all tags applied to those files (i.e., each tag in each tag tree applied to each node, from root to leaf):

<file:bytes> -> #*

Pivot to all tag nodes matching the specified pattern:

  • Pivot from a set of IP addresses to the syn:tag nodes for all tags applied to those IPs that are part of the anonymized infrastructure tag tree:

<inet:ipv4> -> #cno.infra.anon.*

Pivot to tag nodes exactly matching the specified tag:

  • Pivot from a set of nodes to the syn:tag node for #foo.bar (if present on the inbound set of nodes):

<query> -> #foo.bar

Usage Notes:

  • Pivot to all tags ( #* ) and pivot by matching an initial pattern ( #<tag>.* ) will match all tags in the relevant tag trees from the inbound nodes, not just the leaf tags. For example, for an inbound node with tag #foo.bar.baz, #* will return the syn:tag nodes for foo, foo.bar, and foo.bar.baz.

  • When using the asterisk / wildcard ( * ) to match a pattern, the wildcard(s) can be used anywhere within the tag name (value); they are not limited to matching elements within the tag’s dotted namespace. For example, all of the following are valid (though may return different results):

    • -> #aka.thr.*

    • -> #aka.t*

    • -> #a*

    • -> #*thr*

    • -> #*.thr.*

  • The pivot to tags operator does not support pivoting directly to a set of tags specified by a prefix match ( ^ ) or regular expression ( ~ ). However, these operators can be used as part of a subsequent filter operation to further refine the results of the pivot.

Pivot from Tags

Pivot from tags syntax allows you to pivot from a set of syn:tag nodes to the set of nodes that have those tags.

Syntax:

<syn:tag> -> * | <form>

Examples:

  • Pivot to all domains tagged with tags from any of the inbound syn:tag nodes:

<syn:tag> -> inet:fqdn

  • Pivot to all nodes tagged with tags from any of the inbound syn:tag nodes:

<syn:tag> -> *

Usage Notes:

  • In many cases, pivot from tags is functionally equivalent to Lift by Tag (#). That is, the following queries will both return all nodes tagged with #aka.feye.thr.apt1:

    syn:tag=aka.feye.thr.apt1 -> *

    #aka.feye.thr.apt1

    Pivoting from tags is most useful when used in conjunction with Pivot to Tags - that is, taking a set of inbound nodes, pivoting to the syn:tag nodes for any associated tags (pivot to tags), and then pivoting out again to other nodes tagged with some or all of those tags (pivot from tags).

Implicit Pivot Syntax

Pivot operations in Storm can always be executed by explicitly specifying the source and target properties for the pivot. This is referred to as explicit pivot syntax or explicit syntax. For example:

inet:fqdn=vertex.link -> inet:dns:a:fqdn :ipv4 -> inet:ipv4

The above query:

  • lifts the FQDN vertex.link,

  • explicitly pivots from the primary property of the FQDN to any inet:dns:a:fqdn secondary property with the same value, and

  • explicitly pivots from the :ipv4 secondary property of the inet:dns:a nodes to the primary property of any inet:ipv4 nodes with the same value.

Using explicit pivot syntax tells Storm exactly what you want to do; there is no ambiguity in the query. (Explicit syntax may also be useful when first learning Storm to reinforce exactly what navigation is being carried out when you perform a pivot operation.) However, the need to fully specify target properties (using form and property names) and specifically reference source properties (using relative property names) can add overhead (“more typing”) to a Storm query that is not necessary if the query is unambiguous (i.e., based on the inbound and outbound forms).

For this reason, Storm also supports implicit pivot syntax for certain types of pivots. Implicit pivot syntax takes advantage of Synapse’s Type Awareness to “know” which properties can be pivoted to (or from), given the forms that are inbound to and outbound from the pivot operation. In these cases, the source and/or target do not need to be explicitly specified. This allows for more concise Storm syntax in cases where the source and / or target of the pivot is self-evident given the forms used.

Implicit pivot syntax can be used in the following cases where the source and target properties have both the same type AND the same value:

  • Primary to secondary property pivots.

  • Secondary to primary property pivots.

Implicit pivot syntax cannot be used for the following:

  • Primary to primary property pivots (see Raw Pivot Syntax, below)

  • Secondary to secondary property pivots.

  • Pivots between primary and secondary (or secondary and primary) properties with the same value but of different types.

Examples:

Pivot from primary property (<form> = <valu>) to secondary property (<prop> = <pval>) using implicit syntax:

  • Pivot from a set of domains to their associated DNS A records:

Explicit syntax:

<inet:fqdn> -> inet:dns:a:fqdn

Implicit syntax:

<inet:fqdn> -> inet:dns:a

With implicit syntax, the target property :fqdn can be omitted because it is the only logical target given inet:fqdn nodes as the source and inet:dns:a nodes as the target of the pivot.

Note

While the inet:fqdn form has secondary properties that are also of type inet:fqdn (e.g., both :domain and :zone) implicit syntax can only be used to pivot between primary and secondary OR secondary and primary properties, but not both. That is, implicit syntax does not allow you to go from any / all properties of a given type in the source nodes to any / all properties with the same type and value in the target nodes. Because the target of the pivot is inet:dns:a nodes, the only logical target given the inbound nodes is the :fqdn secondary property, which means the only logical source is the primary property of the inet:fqdn.

Pivot from secondary property (<prop> = <pval>) to primary property (<form> = <valu>) using implicit syntax:

  • Pivot from a set of DNS A records to their associated IP addresses:

Explicit syntax:

<inet:dns:a> :ipv4 -> inet:ipv4

Implicit syntax:

<inet:dns:a> -> inet:ipv4

With implicit syntax, the source property :ipv4 can be omitted because it is the only logical source given a set of inet:ipv4 nodes as the target.

Note

Similar to the last example, while the inet:dns:a form has both :ipv4 and :fqdn secondary properties, implicit syntax can only be used to pivot between primary and secondary OR secondary and primary properties, but not both. Because the target of the pivot is inet:ipv4 nodes, the only logical source property is the :ipv4 secondary property of the inet:dns:a node.

Use of multiple implicit pivots:

  • Pivot from a set of domains to their DNS A records and then to the associated IP addresses:

Regular (full) syntax:

<inet:fqdn> -> inet:dns:a:fqdn :ipv4 -> inet:ipv4

Implicit syntax:

<inet:fqdn> -> inet:dns:a -> inet:ipv4

The above example simply combines the previous two examples to illustrate the use of multiple implicit pivot operations in a longer query.

Implicit syntax with multiple target properties:

  • Pivot from a set of domains to the associated DNS MX records:

Implicit syntax:

<inet:fqdn> -> inet:dns:mx

In the example above, given the source and target forms, the logical pivot for implicit syntax is from the primary property of the inbound FQDN to the secondary properties of the DNS MX nodes. However, an inet:dns:mx form has two secondary properties of type inet:fqdn: the mail exchange server (inet:dns:mx:mx) and the domain that uses the MX (inet:dns:mx:fqdn).

Using implicit syntax, Storm wil pivot from the source FQDN(s) to any DNS MX records where the domain matches either of those secondary properties. For example, querying the FQDN google.com will return DNS MX records for Google (i.e., inet:dns:mx:fqdn=google.com) as well as MX records that may use Google as their mail exchange (i.e., inet:dns:mx:mx=google.com).

If you want only one or the other of those types of records, you need to use explicit syntax to specify the target property, i.e.:

<inet:fqdn> -> inet:dns:mx:fqdn

or

<inet:fqdn> -> inet:dns:mx:mx

Implicit syntax with multiple source properties:

  • Pivot from a set of files to their associated SHA256 hashes:

Implicit syntax:

<file:bytes> -> hash:sha256

In the example above, given the source and target forms, the logical pivot for implicit syntax is from the secondary properties of the inbound file to the primary property of the SHA256 nodes. However, a file:bytes form has two secondary properties of type hash:sha256: the file’s SHA256 hash (file:bytes:sha256) and the hash of the file’s rich header data (if the file is a PE executable - file:bytes:mime:pe:richhdr).

Using implicit syntax, Storm will pivot from both source properties (where present) to all of the associated SHA256 nodes - that is, those that match either the :sha256 or :mime:pe:richhdr value.

If you want only one or the other of those types of records, you need to use explicit syntax to specify the source property, i.e.:

<file:bytes> :sha256 -> hash:sha256

or

<file:bytes> :mime:pe:richhdr -> hash:sha256

Raw Pivot Syntax

For certain edge cases, standard Storm pivot syntax (explicit or implicit) is insufficient. In these instances raw pivot syntax acts as a “get out of jail free” card to perform specialized pivot operations. These include:

  • primary-to-primary property pivots;

  • pivots where the value of the target property (primary or secondary) is computed from the input node(s);

  • extramodel pivots.

In raw pivot syntax, the target of the pivot is specified as a Storm query enclosed in curly braces. Raw pivots often involve specifying a variable derived from the inbound node(s) and performing the raw pivot using the variable, though this is not technically required. (See Storm Reference - Advanced - Variables for a discussion of using variables in Storm).

For some raw pivot syntax use cases, you can compose an equivalent Storm query using lift and filter operations. For example:

  • lift a set of nodes;

  • define a variable based on those nodes;

  • lift a second set of nodes using the variable;

  • filter out the original nodes you lifted, thus leaving only the second set of lifted nodes.

However, executing this type of query using raw pivot syntax is slightly more efficient; the Storm query within the raw pivot’s curly braces may still be a lift operation, but performing it inside a raw pivot means you do not have to explicitly drop (filter out) your original nodes. (As with a regular pivot, the inbound nodes are consumed by the pivot operation itself, eliminating the need for the filter.)

As always, these efficiencies may be trivial for smaller queries but can be significant for larger queries.

Syntax:

<query> -> { <query> }

Examples:

  • Pivot from a string (it:dev:str) representing an FQDN to the inet:fqdn node for that FQDN (i.e., pivot between two primary properties of different types).

Standard syntax (no raw pivot, lift / filter only):

<it:dev:str> $fqdn=$node.value() inet:fqdn=$fqdn -it:dev:str

Raw pivot syntax:

<it:dev:str> $fqdn=$node.value() -> { inet:fqdn=$fqdn }