synapse.lib package

Subpackages

• synapse.lib.crypto package
  • Submodules
  • synapse.lib.crypto.coin module
  • synapse.lib.crypto.ecc module
  • synapse.lib.crypto.tinfoil module
• synapse.lib.platforms package
  • Submodules
  • synapse.lib.platforms.common module
  • synapse.lib.platforms.darwin module
  • synapse.lib.platforms.freebsd module
  • synapse.lib.platforms.linux module
  • synapse.lib.platforms.windows module
• synapse.lib.stormlib package
  • Submodules
  • synapse.lib.stormlib.auth module
  • synapse.lib.stormlib.backup module
  • synapse.lib.stormlib.cell module
  • synapse.lib.stormlib.ethereum module
  • synapse.lib.stormlib.hex module
  • synapse.lib.stormlib.imap module
  • synapse.lib.stormlib.infosec module
  • synapse.lib.stormlib.ipv6 module
  • synapse.lib.stormlib.json module
  • synapse.lib.stormlib.log module
  • synapse.lib.stormlib.macro module
  • synapse.lib.stormlib.model module
  • synapse.lib.stormlib.modelext module
  • synapse.lib.stormlib.notifications module
  • synapse.lib.stormlib.oauth module
  • synapse.lib.stormlib.project module
  • synapse.lib.stormlib.scrape module
  • synapse.lib.stormlib.smtp module
  • synapse.lib.stormlib.stix module
  • synapse.lib.stormlib.storm module
  • synapse.lib.stormlib.version module
  • synapse.lib.stormlib.xml module
  • synapse.lib.stormlib.yaml module

Submodules

synapse.lib.agenda module

class synapse.lib.agenda.Agenda

Bases: synapse.lib.base.Base

Organize and execute all the scheduled storm queries in a cortex.

async add(cdef)

Persistently adds an appointment

Parameters

cdef (dict) – Dictionary containing the Cron definition.

Notes

The cron definition may contain the following keys:

creator (str)

Iden of the creating user.

iden (str)

Iden of the appointment.

storm (str)

The Storm query to run.

reqs (Union[None, Dict[TimeUnit, Union[int, Tuple[int]], List[…])

One or more dicts of the fixed aspects of the appointment. dict value may be a single or multiple. May be an empty dict or None.

incunit (Union[None, TimeUnit])

The unit that changes for recurring, or None for non-recurring. It is an error for this value to match a key in reqdict.

incvals (Union[None, int, Iterable[int])

Count of units of incunit or explicit day of week or day of month. Not allowed for incunit == None, required for others (1 would be a typical value)

If the values for req and incvals are both lists, all combinations of all values (the product) are used.

Returns

Packed appointment definition

async delete(iden)

Delete an appointment

async disable(iden)

async enable(iden)

async get(iden)

list()

async mod(iden, query)

Change the query of an appointment

async move(croniden, viewiden)

Move a cronjob from one view to another

async start()

Enable cron jobs to start running, start the scheduler loop

Go through all the appointments, making sure the query is valid, and remove the ones that aren’t. (We can’t evaluate queries until enabled because not all the modules are loaded yet.)

async stop()

Cancel the scheduler loop, and set self.enabled to False.

class synapse.lib.agenda.ApptRec(reqdict, incunit=None, incval=1)

Bases: object

Represents a single element of a single combination of an appointment

nexttime(lastts)

Returns next timestamp that meets requirements, incrementing by (self.incunit * incval) if not increasing, or 0.0 if there are no future matches

pack()

Make ApptRec json/msgpack-friendly

classmethod unpack(val)

Convert from json/msgpack-friendly

class synapse.lib.agenda.TimeUnit(value)

Bases: enum.IntEnum

Unit of time that recurring and required parts of appointments are made of

DAY = 5

DAYOFMONTH = 3

DAYOFWEEK = 4

HOUR = 6

MINUTE = 7

MONTH = 2

NOW = 8

YEAR = 1

classmethod fromString(s)

synapse.lib.aha module

class synapse.lib.aha.AhaApi

Bases: synapse.lib.cell.CellApi

async addAhaSvc(name, info, network=None)

Register a service with the AHA discovery server.

NOTE: In order for the service to remain marked “up” a caller

must maintain the telepath link.

async delAhaSvc(name, network=None)

Remove an AHA service entry.

async genCaCert(network)

async getAhaSvc(name)

Return an AHA service description dictionary for a fully qualified service name.

async getAhaSvcs(network=None)

Yield AHA svcinfo dictionaries.

Parameters

network (str) – Optionally specify a network to filter on.

async getAhaUrls()

async getCaCert(network)

async signHostCsr(csrtext, signas=None, sans=None)

async signUserCsr(csrtext, signas=None)

class synapse.lib.aha.AhaCell

Bases: synapse.lib.cell.Cell

async addAhaSvc(name, info, network=None)

cellapi

alias of synapse.lib.aha.AhaApi

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class.  Used by stemcell.', 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the cell.', 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'defs': {'role': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'$ref': '#/properties/inaugural/defs/rule'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'rule': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'user': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'$ref': '#/properties/inaugural/defs/rule'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}}, 'description': 'Data used to drive configuration of the Cell upon first startup.', 'properties': {'roles': {'items': {'$ref': '#/properties/inaugural/defs/role'}, 'type': 'array'}, 'users': {'items': {'$ref': '#/properties/inaugural/defs/user'}, 'type': 'array'}}, 'type': 'object'}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': False, 'hidedocs': False, 'type': 'string'}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to the cell.  Required for mirroring (on both sides).', 'type': 'boolean'}}

confdefs = {'aha:urls': {'description': 'A list of all available AHA server URLs.', 'items': {'type': 'string'}, 'type': ['string', 'array']}}

async delAhaSvc(name, network=None)

async genCaCert(network)

async getAhaSvc(name)

async getAhaSvcs(network=None)

async getCaCert(network)

async initServiceRuntime()

async initServiceStorage()

async saveCaCert(name, cakey, cacert)

async saveHostCert(name, hostkey, hostcert)

async saveUserCert(name, userkey, usercert)

async setAhaSvcDown(name, linkiden, network=None)

async signHostCsr(csrtext, signas=None, sans=None)

async signUserCsr(csrtext, signas=None)

synapse.lib.ast module

class synapse.lib.ast.AbsProp(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.AbsPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AndCond(kids=())

Bases: synapse.lib.ast.Cond

<cond> and <cond>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

class synapse.lib.ast.ArgvQuery(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

isRuntSafe(runt)

validate(runt)

class synapse.lib.ast.ArrayCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.AstNode(kids=())

Bases: object

Base class for all nodes in the Storm abstract syntax tree.

addKid(astn)

format(depth=0)

getRuntVars(runt)

hasAstClass(clss)

init(core)

isRuntSafe(runt)

iterright()

Yield “rightward” siblings until None.

optimize()

prepare()

repr()

sibling(offs=1)

Return sibling node by relative offset from self.

validate(runt)

class synapse.lib.ast.Bool(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.BreakOper(kids=())

Bases: synapse.lib.ast.AstNode

async run(runt, genr)

class synapse.lib.ast.CallArgs(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.CallKwarg(kids=())

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CallKwargs(kids=())

Bases: synapse.lib.ast.CallArgs

class synapse.lib.ast.CaseEntry(kids=())

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.CatchBlock(kids=())

Bases: synapse.lib.ast.AstNode

async catches(name, runt, path=None)

errvar()

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.CmdOper(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.Cmpr(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.Cond(kids=())

Bases: synapse.lib.ast.Value

A condition that is evaluated to filter nodes.

class synapse.lib.ast.Const(valu, kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

isRuntSafe(runt)

repr()

value()

class synapse.lib.ast.ContinueOper(kids=())

Bases: synapse.lib.ast.AstNode

async run(runt, genr)

class synapse.lib.ast.DollarExpr(kids=())

Bases: synapse.lib.ast.Value

Top level node for $(…) expressions

async compute(runt, path)

class synapse.lib.ast.Edit(kids=())

Bases: synapse.lib.ast.Oper

class synapse.lib.ast.EditEdgeAdd(kids=(), n2=False)

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditEdgeDel(kids=(), n2=False)

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditNodeAdd(kids=())

Bases: synapse.lib.ast.Edit

async addFromPath(form, runt, path)

Add a node using the context from path.

NOTE: CALLER MUST CHECK PERMS

prepare()

async run(runt, genr)

class synapse.lib.ast.EditParens(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditPropDel(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditPropSet(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditTagAdd(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditTagDel(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EditTagPropDel(kids=())

Bases: synapse.lib.ast.Edit

[ -#foo.bar:baz ]

async run(runt, genr)

class synapse.lib.ast.EditTagPropSet(kids=())

Bases: synapse.lib.ast.Edit

[ #foo.bar:baz=10 ]

async run(runt, genr)

class synapse.lib.ast.EditUnivDel(kids=())

Bases: synapse.lib.ast.Edit

async run(runt, genr)

class synapse.lib.ast.EmbedQuery(valu, kids=())

Bases: synapse.lib.ast.Const

async compute(runt, path)

getRuntVars(runt)

validate(runt)

class synapse.lib.ast.Emit(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.ExprAndNode(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.ExprDict(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.ExprList(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.ExprNode(kids=())

Bases: synapse.lib.ast.Value

A binary (i.e. two argument) expression node

async compute(runt, path)

prepare()

class synapse.lib.ast.ExprOrNode(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.FiltByArray(kids=())

Bases: synapse.lib.ast.FiltOper

+:foo*[^=visi]

class synapse.lib.ast.FiltOper(kids=())

Bases: synapse.lib.ast.Oper

async getLiftHints(runt, path)

async run(runt, genr)

class synapse.lib.ast.FiniBlock(kids=())

Bases: synapse.lib.ast.AstNode

An AST node that runs only once after all nodes have been consumed.

Example

Using a fini block:

fini {
   // stuff here runs *once* after the last node yield (even if there are no nodes)
}

Notes

A fini block must be runtsafe.

async run(runt, genr)

class synapse.lib.ast.ForLoop(kids=())

Bases: synapse.lib.ast.Oper

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.FormName(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.FormPivot(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> foo:bar

async run(runt, genr)

class synapse.lib.ast.FormTagProp(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.FuncArgs(kids=())

Bases: synapse.lib.ast.AstNode

Represents the function arguments in a function definition

async compute(runt, path)

class synapse.lib.ast.FuncCall(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.Function(kids=())

Bases: synapse.lib.ast.AstNode

( name, args, body )

// use args/kwargs syntax function bar(x, v=$(30)) { }

# we auto-detect the behavior of the target function

# return a value function bar(x, y) { return ($(x + y)) }

# a function that produces nodes function bar(x, y) { [ baz:faz=(x, y) ] }

$foo = $bar(10, v=20)

async callfunc(runt, argdefs, args, kwargs)

Execute a function call using the given runtime.

This function may return a value / generator / async generator

getRuntVars(runt)

isRuntSafe(runt)

prepare()

async run(runt, genr)

validate(runt)

class synapse.lib.ast.HasAbsPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.HasRelPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

async hasProp(node, runt, name)

class synapse.lib.ast.HasTagPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.IfClause(kids=())

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.IfStmt(kids=())

Bases: synapse.lib.ast.Oper

prepare()

async run(runt, genr)

class synapse.lib.ast.InitBlock(kids=())

Bases: synapse.lib.ast.AstNode

An AST node that runs only once before yielding nodes.

Example

Using a init block:

init {
    // stuff here runs *once* before the first node yield (even if there are no nodes)
}

async run(runt, genr)

class synapse.lib.ast.LiftByArray(kids=())

Bases: synapse.lib.ast.LiftOper

:prop*[range=(200, 400)]

async lift(runt, path)

class synapse.lib.ast.LiftFormTag(kids=())

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)

class synapse.lib.ast.LiftFormTagProp(kids=())

Bases: synapse.lib.ast.LiftOper

hehe:haha#foo.bar:baz [ = x ]

async lift(runt, path)

class synapse.lib.ast.LiftOper(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.LiftProp(kids=())

Bases: synapse.lib.ast.LiftOper

async getRightHints(runt, path)

async lift(runt, path)

class synapse.lib.ast.LiftPropBy(kids=())

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)

class synapse.lib.ast.LiftTag(kids=())

Bases: synapse.lib.ast.LiftOper

async lift(runt, path)

class synapse.lib.ast.LiftTagProp(kids=())

Bases: synapse.lib.ast.LiftOper

#foo.bar:baz [ = x ]

async lift(runt, path)

class synapse.lib.ast.LiftTagTag(kids=())

Bases: synapse.lib.ast.LiftOper

##foo.bar

async lift(runt, path)

class synapse.lib.ast.List(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

repr()

class synapse.lib.ast.Lookup(kids, autoadd=False)

Bases: synapse.lib.ast.Query

When storm input mode is “lookup”

async run(runt, genr)

class synapse.lib.ast.N1Walk(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

async walkNodeEdges(runt, node, verb=None)

class synapse.lib.ast.N1WalkNPivo(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOut

async run(runt, genr)

class synapse.lib.ast.N2Walk(kids=())

Bases: synapse.lib.ast.N1Walk

async walkNodeEdges(runt, node, verb=None)

class synapse.lib.ast.N2WalkNPivo(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotIn

async run(runt, genr)

class synapse.lib.ast.NotCond(kids=())

Bases: synapse.lib.ast.Cond

not <cond>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.Oper(kids=())

Bases: synapse.lib.ast.AstNode

class synapse.lib.ast.OrCond(kids=())

Bases: synapse.lib.ast.Cond

<cond> or <cond>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.PivotIn(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

<- *

async getPivsIn(runt, node, path)

async run(runt, genr)

class synapse.lib.ast.PivotInFrom(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

<- foo:edge

async run(runt, genr)

class synapse.lib.ast.PivotOper(kids=(), isjoin=False)

Bases: synapse.lib.ast.Oper

repr()

class synapse.lib.ast.PivotOut(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> *

async getPivsOut(runt, node, path)

async run(runt, genr)

class synapse.lib.ast.PivotToTags(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> # pivot to all leaf tag nodes -> #* pivot to all tag nodes -> #cno.* pivot to all tag nodes which match cno.* -> #foo.bar pivot to the tag node foo.bar if present

async run(runt, genr)

class synapse.lib.ast.PropName(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.PropPivot(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

:foo -> bar:foo

async run(runt, genr)

class synapse.lib.ast.PropPivotOut(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

:prop -> *

async run(runt, genr)

class synapse.lib.ast.PropValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

async getPropAndValu(runt, path)

isRuntSafe(valu)

prepare()

class synapse.lib.ast.Query(kids=())

Bases: synapse.lib.ast.AstNode

async iterNodePaths(runt, genr=None)

async run(runt, genr)

class synapse.lib.ast.RawPivot(kids=(), isjoin=False)

Bases: synapse.lib.ast.PivotOper

-> { <varsfrompath> }

async run(runt, genr)

class synapse.lib.ast.RelProp(kids=())

Bases: synapse.lib.ast.PropName

class synapse.lib.ast.RelPropCond(kids=())

Bases: synapse.lib.ast.Cond

(:foo:bar or .univ) <cmpr> <value>

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

class synapse.lib.ast.RelPropValue(kids=())

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Return(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.SetItemOper(kids=())

Bases: synapse.lib.ast.Oper

$foo.bar = baz $foo.”bar baz” = faz $foo.$bar = baz

async run(runt, genr)

class synapse.lib.ast.SetVarOper(kids=())

Bases: synapse.lib.ast.Oper

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.Stop(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.SubGraph(rules)

Bases: object

An Oper like object which generates a subgraph.

Notes

The rules format for the subgraph is shaped like the following:

rules = {

    'degrees': 1,

    'edges': True,
    'filterinput': True,
    'yieldfiltered': False,

    'filters': [
        '-(#foo or #bar)',
        '-(foo:bar or baz:faz)',
    ],

    'pivots': [
        '-> * | limit 100',
        '<- * | limit 100',
    ]

    'forms': {

        'inet:fqdn':{
            'filters': [],
            'pivots': [],
        }

        '*': {
            'filters': [],
            'pivots': [],
        },
    },
}

Nodes which were original seeds have path.meta(‘graph:seed’).

All nodes have path.meta(‘edges’) which is a list of (iden, info) tuples.

async omit(runt, node)

async pivots(runt, node, path)

async run(runt, genr)

class synapse.lib.ast.SubQuery(kids=())

Bases: synapse.lib.ast.Oper

async compute(runt, path)

Use subquery as a value. It is error if the subquery used in this way doesn’t yield exactly one node or has a return statement.

Its value is the primary property of the node yielded, or the returned value.

async compute_array(runt, path)

Use subquery as an array.

async inline(runt, genr)

Operate subquery as if it were inlined

async run(runt, genr)

class synapse.lib.ast.SubqCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.SwitchCase(kids=())

Bases: synapse.lib.ast.Oper

prepare()

async run(runt, genr)

class synapse.lib.ast.TagCond(kids=())

Bases: synapse.lib.ast.Cond

#foo.bar

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

class synapse.lib.ast.TagMatch(kids=())

Bases: synapse.lib.ast.TagName

Like TagName, but can have asterisks

hasglob()

class synapse.lib.ast.TagName(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

prepare()

class synapse.lib.ast.TagProp(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.TagPropCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagPropValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.TagValuCond(kids=())

Bases: synapse.lib.ast.Cond

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

class synapse.lib.ast.TagValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

isRuntSafe(runt)

class synapse.lib.ast.TryCatch(kids=())

Bases: synapse.lib.ast.AstNode

async getCatchBlock(name, runt, path=None)

async getErrValu(e)

async run(runt, genr)

class synapse.lib.ast.UnaryExprNode(kids=())

Bases: synapse.lib.ast.Value

A unary (i.e. single-argument) expression node

async compute(runt, path)

prepare()

class synapse.lib.ast.UnivProp(kids=())

Bases: synapse.lib.ast.RelProp

async compute(runt, path)

class synapse.lib.ast.UnivPropValue(kids=())

Bases: synapse.lib.ast.PropValue

class synapse.lib.ast.Value(kids=())

Bases: synapse.lib.ast.AstNode

The base class for all values and value expressions.

async compute(runt, path)

async getCondEval(runt)

Return a function that may be used to evaluate the boolean truth of the value expression using a runtime and optional node path.

async getLiftHints(runt, path)

isRuntSafe(runt)

class synapse.lib.ast.VarDeref(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

class synapse.lib.ast.VarEvalOper(kids=())

Bases: synapse.lib.ast.Oper

Facilitate a stand-alone operator that evaluates a var. $foo.bar(“baz”)

async run(runt, genr)

class synapse.lib.ast.VarList(valu, kids=())

Bases: synapse.lib.ast.Const

class synapse.lib.ast.VarListSetOper(kids=())

Bases: synapse.lib.ast.Oper

getRuntVars(runt)

async run(runt, genr)

class synapse.lib.ast.VarValue(kids=())

Bases: synapse.lib.ast.Value

async compute(runt, path)

isRuntSafe(runt)

prepare()

validate(runt)

class synapse.lib.ast.WhileLoop(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

class synapse.lib.ast.YieldValu(kids=())

Bases: synapse.lib.ast.Oper

async run(runt, genr)

async yieldFromValu(runt, valu)

async synapse.lib.ast.expr_add(x, y)

async synapse.lib.ast.expr_div(x, y)

async synapse.lib.ast.expr_eq(x, y)

async synapse.lib.ast.expr_ge(x, y)

async synapse.lib.ast.expr_gt(x, y)

async synapse.lib.ast.expr_le(x, y)

async synapse.lib.ast.expr_lt(x, y)

async synapse.lib.ast.expr_mul(x, y)

async synapse.lib.ast.expr_ne(x, y)

async synapse.lib.ast.expr_not(x)

async synapse.lib.ast.expr_prefix(x, y)

async synapse.lib.ast.expr_re(x, y)

async synapse.lib.ast.expr_sub(x, y)

synapse.lib.ast.parseNumber(x)

async synapse.lib.ast.pullone(genr)

synapse.lib.autodoc module

class synapse.lib.autodoc.RstHelp

Bases: object

addHead(name, lvl=0, link=None)

addLines(*lines)

getRstText()

synapse.lib.autodoc.docStormTypes(page, docinfo, linkprefix, islib=False, lvl=1, known_types=None, types_prefix=None, types_suffix=None)

Process a list of StormTypes doc information to add them to a a RstHelp object.

Notes

This will create internal hyperlink link targets for each header item. The link prefix string must be given with the linkprefix argument.

Parameters

• page (RstHelp) – The RST page to add .

• docinfo (dict) – A Stormtypes Doc.

• linkprefix (str) – The RST link prefix string to use.

• islib (bool) – Treat the data as a library. This will preface the header and attribute values with $ and use full paths for attributes.

• lvl (int) – The base header level to use when adding headers to the page.

Returns

None

synapse.lib.autodoc.genCallsig(rtype)

synapse.lib.autodoc.getArgLines(rtype)

synapse.lib.autodoc.getLink(sname, linkprefix, ref=False, suffix=None)

synapse.lib.autodoc.getReturnLines(rtype, known_types=None, types_prefix=None, suffix=None, isstor=False)

synapse.lib.autodoc.getRtypeStr(rtype, known_types, types_prefix, suffix)

synapse.lib.autodoc.ljuster(ilines)

Helper to lstrip lines of whitespace an appropriate amount.

synapse.lib.autodoc.prepareRstLines(doc)

Prepare a desc string for RST lines.

synapse.lib.autodoc.scrubLines(lines)

Remove any empty lines until we encounter non-empty linee

synapse.lib.base module

class synapse.lib.base.Base

Bases: object

Base class for Synapse objects.

Acts as an observable, enables async init and fini.

Example

class Foo(Base):

async def __anit__(self, x, y):

await Base.__anit__(self)

await stuff(x, y)

foo = await Foo.anit(10)

Note

One should not create instances directly via its initializer, i.e. Base(). One shall always use the class method anit.

async addSignalHandlers()

Register SIGTERM/SIGINT signal handlers with the ioloop to fini this object.

async classmethod anit(*args, **kwargs)

async dist(mesg)

Distribute an existing event tuple.

Parameters

mesg ((str,dict)) – An event tuple.

Example

await base.dist( (‘foo’,{‘bar’:’baz’}) )

async enter_context(item)

Modeled on Python’s contextlib.ExitStack.enter_context. Enters a new context manager and adds its __exit__() and __aexit__ method to its onfini handlers.

Returns

The result of item’s own __aenter__ or __enter__() method.

async fini()

Shut down the object and notify any onfini() coroutines.

Returns

Remaining ref count

async fire(evtname, **info)

Fire the given event name on the Base. Returns a list of the return values of each callback.

Example

for ret in d.fire(‘woot’,foo=’asdf’):

print(‘got: %r’ % (ret,))

incref()

Increment the reference count for this base. This API may be optionally used to control fini().

link(func)

Add a callback function to receive all events.

Example

base1 = Base() base2 = Base()

base1.link( base2.dist )

# all events on base1 are also propagated on base2

async main()

Helper function to setup signal handlers for this base as the main object. ( use base.waitfini() to block )

NOTE: This API may only be used when the ioloop is also the main thread.

off(evnt, func)

Remove a previously registered event handler function.

Example

base.off( ‘foo’, onFooFunc )

on(evnt, func, base=None)

Add an base function callback for a specific event with optional filtering. If the function returns a coroutine, it will be awaited.

Parameters

• evnt (str) – An event name

• func (function) – A callback function to receive event tufo

Examples

Add a callback function and fire it:

async def baz(event):

x = event[1].get(‘x’) y = event[1].get(‘y’) return x + y

d.on(‘foo’, baz)

# this fire triggers baz… await d.fire(‘foo’, x=10, y=20)

Return type

None

onWith(evnt, func)

A context manager which can be used to add a callback and remove it when using a with statement.

Parameters

• evnt (str) – An event name

• func (function) – A callback function to receive event tufo

onfini(func)

Add a function/coroutine/Base to be called on fini().

async postAnit()

Method called after self.__anit__() has completed, but before anit() returns the object to the caller.

schedCallSafe(func, *args, **kwargs)

Schedule a function to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on the function completion.

Parameters

• func –

• *args –

• **kwargs –

Notes

This method may be called from outside of the event loop on a different thread.

Returns

A Future representing the eventual function execution.

Return type

concurrent.futures.Future

schedCoro(coro)

Schedules a free-running coroutine to run on this base’s event loop. Kills the coroutine if Base is fini’d. It does not pend on coroutine completion.

Precondition:

This function is not threadsafe and must be run on the Base’s event loop

Returns

An asyncio.Task object.

Return type

asyncio.Task

schedCoroSafe(coro)

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on.

This function does not pend on coroutine completion.

Notes

This method may be run outside the event loop on a different thread.

Returns

A Future representing the eventual coroutine execution.

Return type

concurrent.futures.Future

schedCoroSafePend(coro)

Schedules a coroutine to run as soon as possible on the same event loop that this Base is running on

Note

This method may not be run inside an event loop

unlink(func)

Remove a callback function previously added with link()

Example

base.unlink( callback )

waiter(count, *names)

Construct and return a new Waiter for events on this base.

Example

# wait up to 3 seconds for 10 foo:bar events…

waiter = base.waiter(10,’foo:bar’)

# .. fire thread that will cause foo:bar events

events = await waiter.wait(timeout=3)

if events == None:

# handle the timeout case…

for event in events:

# parse the events if you need…

NOTE: use with caution… it’s easy to accidentally construct

race conditions with this mechanism ;)

async waitfini(timeout=None)

Wait for the base to fini()

Returns

None if timed out, True if fini happened

Example

base.waitfini(timeout=30)

class synapse.lib.base.BaseRef

Bases: synapse.lib.base.Base

An object for managing multiple Base instances by name.

async gen(name)

Atomically get/gen a Base and incref. (requires ctor during BaseRef init)

Parameters

name (str) – The name/iden of the Base instance.

get(name)

Retrieve a Base instance by name.

Parameters

name (str) – The name/iden of the Base

Returns

The Base instance (or None)

Return type

(Base)

items()

pop(name)

Remove and return a Base from the BaseRef.

Parameters

name (str) – The name/iden of the Base instance

Returns

The named base ( or None )

Return type

(Base)

put(name, base)

Add a Base (or sub-class) to the BaseRef by name.

Parameters

• name (str) – The name/iden of the Base

• base (Base) – The Base instance

Returns

(None)

vals()

class synapse.lib.base.Waiter(base, count, *names)

Bases: object

A helper to wait for a given number of events on a Base.

fini()

async wait(timeout=None)

Wait for the required number of events and return them or None on timeout.

Example

evnts = waiter.wait(timeout=30)

if evnts == None:

handleTimedOut() return

for evnt in evnts:

doStuff(evnt)

async synapse.lib.base.main(coro)

async synapse.lib.base.schedGenr(genr, maxsize=100)

Schedule a generator to run on a separate task and yield results to this task (pipelined generator).

synapse.lib.boss module

class synapse.lib.boss.Boss

Bases: synapse.lib.base.Base

An object to track “promoted” async tasks.

async execute(coro, name, user, info=None)

Create a synapse task from the given coroutine.

get(iden)

async promote(name, user, info=None)

Promote the currently running task.

Parameters

• name (str) – The name of the task.

• user – The User who owns the task.

• info – An optional information dictionary containing information about the task.

Returns

The Synapse Task object.

Return type

s_task.Task

ps()

synapse.lib.cache module

A few speed optimized (lockless) cache helpers. Use carefully.

class synapse.lib.cache.FixedCache(callback, size=10000)

Bases: object

async aget(key)

clear()

get(key)

pop(key)

put(key, val)

class synapse.lib.cache.LruDict(size=10000)

Bases: collections.abc.MutableMapping

Maintains the last n accessed keys

get(key, default=None)

Note: we override default impl from parent to avoid costly KeyError

items() → a set-like object providing a view on D's items

values() → an object providing a view on D's values

class synapse.lib.cache.TagGlobs

Bases: object

An object that manages multiple tag globs and values for caching.

add(name, valu, base=None)

get(name)

rem(name, valu)

synapse.lib.cache.getTagGlobRegx(name)

synapse.lib.cache.memoize(size=16384)

synapse.lib.cache.memoizemethod(size=16384)

A version of memoize that doesn’t cause GC cycles when applied to a method.

synapse.lib.cache.regexizeTagGlob(tag)

Returns

a regular expression string with ** and * interpreted as tag globs

Precondition:

tag is a valid tagmatch

Notes

A single asterisk will replace exactly one dot-delimited component of a tag A double asterisk will replace one or more of any character.

The returned string does not contain a starting ‘^’ or trailing ‘$’.

synapse.lib.cell module

class synapse.lib.cell.Cell

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

A Cell() implements a synapse micro-service.

A Cell has 5 phases of startup:

1. Universal cell data structures

2. Service specific storage/data (pre-nexs)

3. Nexus subsystem initialization

4. Service specific startup (with nexus)

5. Networking and mirror services

BACKUP_SPAWN_TIMEOUT = 60.0

COMMIT = ''

VERSION = (2, 92, 0)

VERSTRING = '2.92.0'

addActiveCoro(func, iden=None, base=None)

Add a function callback to be run as a coroutine when the Cell is active.

Parameters

• func (coroutine function) – The function run as a coroutine.

• iden (str) – The iden to use for the coroutine.

• base (Optional[Base]) – if present, this active coro will be fini’d when the base is fini’d

Returns

A GUID string that identifies the coroutine for delActiveCoro()

Return type

str

Note

This will re-fire the coroutine if it exits and the Cell is still active.

addHealthFunc(func)

Register a callback function to get a HealthCheck object.

addHttpApi(path, ctor, info)

async addHttpsPort(port, host='0.0.0.0', sslctx=None)

async addRole(name)

async addRoleRule(iden, rule, indx=None, gateiden=None)

async addUser(name, passwd=None, email=None, iden=None)

async addUserRole(useriden, roleiden)

async addUserRule(iden, rule, indx=None, gateiden=None)

cellapi

alias of synapse.lib.cell.CellApi

confbase = {'_log_conf': {'description': 'Opaque structure used for logging by spawned processes.', 'hideconf': True, 'type': 'object'}, 'aha:admin': {'description': 'An AHA client certificate CN to register as a local admin user.', 'type': 'string'}, 'aha:leader': {'description': 'The AHA service name to claim as the active instance of a storm service.', 'type': 'string'}, 'aha:name': {'description': 'The name of the cell service in the aha service registry.', 'type': 'string'}, 'aha:network': {'description': 'The AHA service network. This makes aha:name/aha:leader relative names.', 'type': 'string'}, 'aha:registry': {'description': 'The telepath URL of the aha service registry.', 'items': {'type': 'string'}, 'type': ['string', 'array']}, 'aha:svcinfo': {'description': 'An AHA svcinfo object. If set, this overrides self discovered Aha service information.', 'properties': {'urlinfo': {'properties': {'host': {'type': 'string'}, 'port': {'type': 'integer'}, 'schema': {'type': 'string'}}, 'required': ('host', 'port', 'scheme'), 'type': 'object'}}, 'required': ('urlinfo',), 'type': 'object'}, 'auth:anon': {'description': 'Allow anonymous telepath access by mapping to the given user name.', 'type': 'string'}, 'auth:conf': {'description': 'Extended configuration to be used by an alternate auth constructor.', 'hideconf': True, 'type': 'object'}, 'auth:ctor': {'description': 'Allow the construction of the cell auth object to be hooked at runtime.', 'hideconf': True, 'type': 'string'}, 'auth:passwd': {'description': 'Set to <passwd> (local only) to bootstrap the root user password.', 'type': 'string'}, 'backup:dir': {'description': 'A directory outside the service directory where backups will be saved. Defaults to ./backups in the service storage directory.', 'type': 'string'}, 'cell:ctor': {'description': 'An optional python path to the Cell class.  Used by stemcell.', 'type': 'string'}, 'cell:guid': {'description': 'An optional hard-coded GUID to store as the permanent GUID for the cell.', 'type': 'string'}, 'dmon:listen': {'description': 'A config-driven way to specify the telepath bind URL.', 'type': ['string', 'null']}, 'https:headers': {'description': 'Headers to add to all HTTPS server responses.', 'hidecmdl': True, 'type': 'object'}, 'https:port': {'description': 'A config-driven way to specify the HTTPS port.', 'type': ['integer', 'null']}, 'inaugural': {'defs': {'role': {'additionalProperties': False, 'properties': {'name': {'pattern': '^(?!all$).+$', 'type': 'string'}, 'rules': {'items': {'$ref': '#/properties/inaugural/defs/rule'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}, 'rule': {'items': [{'type': 'boolean'}, {'type': 'array', 'items': {'type': 'string'}}], 'maxItems': 2, 'minItems': 2, 'type': 'array'}, 'user': {'additionalProperties': False, 'properties': {'admin': {'default': False, 'type': 'boolean'}, 'email': {'type': 'string'}, 'name': {'pattern': '^(?!root$).+$', 'type': 'string'}, 'roles': {'items': {'type': 'string'}, 'type': 'array'}, 'rules': {'items': {'$ref': '#/properties/inaugural/defs/rule'}, 'type': 'array'}}, 'required': ['name'], 'type': 'object'}}, 'description': 'Data used to drive configuration of the Cell upon first startup.', 'properties': {'roles': {'items': {'$ref': '#/properties/inaugural/defs/role'}, 'type': 'array'}, 'users': {'items': {'$ref': '#/properties/inaugural/defs/user'}, 'type': 'array'}}, 'type': 'object'}, 'mirror': {'description': 'A telepath URL for our upstream mirror (we must be a backup!).', 'hidecmdl': True, 'hidedocs': True, 'type': 'string'}, 'nexslog:async': {'default': False, 'description': '(Experimental) Map the nexus log LMDB instance with map_async=True.', 'type': 'boolean'}, 'nexslog:en': {'default': False, 'description': 'Record all changes to the cell.  Required for mirroring (on both sides).', 'type': 'boolean'}}

confdefs = {}

async cullNexsLog(offs)

async delActiveCoro(iden)

Remove an Active coroutine previously added with addActiveCoro().

Parameters

iden (str) – The iden returned by addActiveCoro()

async delBackup(name)

async delRole(iden)

async delRoleRule(iden, rule, gateiden=None)

async delUser(iden)

async delUserRole(useriden, roleiden)

async delUserRule(iden, rule, gateiden=None)

async dyncall(iden, todo, gatekeys=())

async dyniter(iden, todo, gatekeys=())

async classmethod execmain(argv, outp=None)

The main entry point for running the Cell as an application.

Parameters

• argv (list) – A list of command line arguments to launch the Cell with.

• outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This coroutine waits until the Cell is fini’d or a SIGINT/SIGTERM signal is sent to the process.

Returns

None.

async fini()

Fini override that ensures locking teardown order.

async genHttpSess(iden)

classmethod getArgParser(conf=None)

Get an argparse.ArgumentParser for the Cell.

Parameters

conf (s_config.Config) – Optional, a Config object which

Notes

Boot time configuration data is placed in the argument group called config. This adds default dirn, --telepath, --https and --name arguements to the argparser instance. Configuration values which have the hideconf or hidecmdl value set to True are not added to the argparser instance.

Returns

A ArgumentParser for the Cell.

Return type

argparse.ArgumentParser

async getAuthGate(iden)

async getAuthGates()

async getAuthRoles()

async getAuthUsers(archived=False)

async getBackupInfo()

Gets information about recent backup activity

async getBackups()

async getCellApi(link, user, path)

getCellIden()

async getCellInfo()

Return metadata specific for the Cell.

Notes

By default, this function returns information about the base Cell implementation, which reflects the base information in the Synapse Cell.

It is expected that implementers override the following Class attributes in order to provide meaningful version information:

COMMIT - A Git Commit VERSION - A Version tuple. VERSTRING - A Version string.

Returns

A Dictionary of metadata.

Return type

Dict

getCellNexsRoot()

classmethod getCellType()

async getConfOpt(name)

async getDmonSessions()

classmethod getEnvPrefix()

async getHealthCheck()

async getHiveKey(path)

Get the value of a key in the cell default hive

async getHiveKeys(path)

Return a list of (name, value) tuples for nodes under the path.

getLocalProxy(share='*', user='root')

getLocalUrl(share='*', user='root')

async getLogExtra(**kwargs)

Get an extra dictionary for structured logging which can be used as a extra argument for loggers.

Parameters

**kwargs – Additional key/value items to add to the log.

Returns

A dictionary

Return type

Dict

async getNexsIndx()

async getNexusChanges(offs)

async getRoleDef(iden)

async getRoleDefByName(name)

async getRoleDefs()

async getSystemInfo()

Get info about the system in which the cell is running

Returns

• volsize - Volume where cell is running total space

• volfree - Volume where cell is running free space

• backupvolsize - Backup directory volume total space

• backupvolfree - Backup directory volume free space

• cellstarttime - Cell start time in epoch milliseconds

• celluptime - Cell uptime in milliseconds

• cellrealdisk - Cell’s use of disk, equivalent to du

• cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l

• osversion - OS version/architecture

• pyversion - Python version

• totalmem - Total memory in the system

• availmem - Available memory in the system

• cpucount - Number of CPUs on system

Return type

A dictionary with the following keys. All size values are in bytes

async getTeleApi(link, mesg, path)

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

async getUserDef(iden)

async getUserDefByName(name)

async getUserDefs()

getUserName(iden, defv='<unknown>')

Translate the user iden to a user name.

async getUserProfInfo(iden, name)

async getUserProfile(iden)

classmethod initCellConf()

Create a Config object for the Cell.

Notes

The Config object has a envar_prefix set according to the results of cls.getEnvPrefix().

Returns

A Config helper object.

Return type

s_config.Config

async classmethod initFromArgv(argv, outp=None)

Cell launcher which does automatic argument parsing, environment variable resolution and Cell creation.

Parameters

• argv (list) – A list of command line arguments to launch the Cell with.

• outp (s_ouput.OutPut) – Optional, an output object. No longer used in the default implementation.

Notes

This does the following items:

• Create a Config object from the Cell class.

• Creates an Argument Parser from the Cell class and Config object.

• Parses the provided arguments.

• Loads configuration data from the parsed options and environment variables.

• Sets logging for the process.

• Creates the Cell from the Cell Ctor.

• Adds a Telepath listener, HTTPs port listeners and Telepath share names.

• Returns the Cell.

Returns

This returns an instance of the Cell.

Return type

Cell

async initNexusSubsystem()

async initServiceActive()

async initServiceNetwork()

async initServicePassive()

async initServiceRuntime()

async initServiceStorage()

initSslCtx(certpath, keypath)

async isCellActive()

async isUserAllowed(iden, perm, gateiden=None)

async iterBackupArchive(name, user)

async iterNewBackupArchive(user, name=None, remove=False)

async kill(user, iden)

async listHiveKey(path=None)

async loadHiveTree(tree, path=(), trim=False)

Note: this is for expert emergency use only.

async popHiveKey(path)

Remove and return the value of a key in the cell default hive.

Note: this is for expert emergency use only.

async promote()

Transform this cell from a passive follower to an active cell that writes changes locally.

async ps(user)

async reqGateKeys(gatekeys)

async rotateNexsLog()

async runBackup(name=None, wait=True)

async saveHiveTree(path=())

async setCellActive(active)

async setHiveKey(path, valu)

Set or change the value of a key in the cell default hive

async setNexsIndx(indx)

async setRoleName(iden, name)

async setRoleRules(iden, rules, gateiden=None)

async setUserAdmin(iden, admin, gateiden=None)

async setUserArchived(iden, archived)

async setUserEmail(useriden, email)

async setUserLocked(iden, locked)

async setUserName(useriden, name)

async setUserPasswd(iden, passwd)

async setUserProfInfo(iden, name, valu)

async setUserRoles(useriden, roleidens)

async setUserRules(iden, rules, gateiden=None)

async sync()

no-op mutable for testing purposes. If I am follower, when this returns, I have received and applied all the writes that occurred on the leader before this call.

async trimNexsLog(consumers=None, timeout=30)

async tryUserPasswd(name, passwd)

async waitNexsOffs(offs, timeout=None)

class synapse.lib.cell.CellApi

Bases: synapse.lib.base.Base

addAuthRole(name)

addAuthRule(name, rule, indx=None, gateiden=None)

addRole(name)

addRoleRule(iden, rule, indx=None, gateiden=None)

addUser(name, passwd=None, email=None, iden=None)

addUserRole(useriden, roleiden)

addUserRule(iden, rule, indx=None, gateiden=None)

async allowed(perm, default=None)

Check if the user has the requested permission.

Parameters

• perm – permission path components to check

• default – Value returned if no value stored

Examples

Form a path and check the permission from a remote proxy:

perm = ('node', 'add', 'inet:ipv4')
allowed = await prox.allowed(perm)
if allowed:
    dostuff()

Returns

True if the user has permission, False if explicitly denied, None if no entry

Return type

Optional[bool]

cullNexsLog(offs)

Remove Nexus log entries up to (and including) the given offset.

Note

If there are consumers of this cell’s nexus log they must be caught up to at least the offs argument before culling.

Only rotated logs where the last index is less than the provided offset will be removed from disk.

Parameters

offs (int) – The offset to remove entries up to.

Returns

Whether the cull was executed

Return type

bool

delAuthRole(name)

delAuthRule(name, rule, gateiden=None)

delAuthUser(name)

delBackup(name)

Delete a backup by name.

Parameters

name (str) – The name of the backup to delete.

delRole(iden)

delRoleRule(iden, rule, gateiden=None)

delUser(iden)

delUserRole(useriden, roleiden)

delUserRule(iden, rule, gateiden=None)

dyncall(iden, todo, gatekeys=())

dyniter(iden, todo, gatekeys=())

getAuthGate(iden)

getAuthGates()

getAuthInfo(name)

getAuthRoles()

getAuthUsers(archived=False)

Parameters

archived (bool) – If true, list all users, else list non-archived users

getBackupInfo()

Get information about recent backup activity.

Returns

• currduration - If backup currently running, time in ms since backup started, otherwise None

• laststart - Last time (in epoch milliseconds) a backup started

• lastend - Last time (in epoch milliseconds) a backup ended

• lastduration - How long last backup took in ms

• lastsize - Disk usage of last backup completed

• lastupload - Time a backup was last completed being uploaded via iter(New)BackupArchive

• lastexception - Tuple of exception information if last backup failed, otherwise None

Return type

(dict) It has the following keys

Note: these statistics are not persistent, i.e. they are not preserved between cell restarts.

getBackups()

Retrieve a list of backups.

Returns

A list of backup names.

Return type

list[str]

getCellIden()

async getCellInfo()

getCellType()

getCellUser()

getDiagInfo()

getDmonSessions()

async getHealthCheck()

getHiveKey(path)

getHiveKeys(path)

getNexsIndx()

getNexusChanges(offs)

getRoleDef(iden)

getRoleDefByName(name)

getRoleDefs()

async getRoleInfo(name)

getSystemInfo()

Get info about the system in which the cell is running

Returns

• volsize - Volume where cell is running total space

• volfree - Volume where cell is running free space

• backupvolsize - Backup directory volume total space

• backupvolfree - Backup directory volume free space

• celluptime - Cell uptime in milliseconds

• cellrealdisk - Cell’s use of disk, equivalent to du

• cellapprdisk - Cell’s apparent use of disk, equivalent to ls -l

• osversion - OS version/architecture

• pyversion - Python version

• totalmem - Total memory in the system

• availmem - Available memory in the system

Return type

A dictionary with the following keys. All size values are in bytes

getUserDef(iden)

getUserDefByName(name)

getUserDefs()

async getUserInfo(name)

getUserProfInfo(iden, name)

getUserProfile(iden)

async initCellApi()

async isCellActive()

Returns True if the cell is an active/leader cell.

isUserAllowed(iden, perm, gateiden=None)

issue(nexsiden: str, event: str, args, kwargs, meta=None)

iterBackupArchive(name)

Retrieve a backup by name as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters

name (str) – The name of the backup to retrieve.

iterNewBackupArchive(name=None, remove=False)

Run a new backup and return it as a compressed stream of bytes.

Note: Compression and streaming will occur from a separate process.

Parameters

• name (str) – The name of the backup to retrieve.

• remove (bool) – Delete the backup after streaming.

async kill(iden)

listHiveKey(path=None)

popHiveKey(path)

promote()

async ps()

rotateNexsLog()

Rotate the Nexus log at the current offset.

Returns

The starting index of the active Nexus log

Return type

int

runBackup(name=None, wait=True)

Run a new backup.

Parameters

• name (str) – The optional name of the backup.

• wait (bool) – On True, wait for backup to complete before returning.

Returns

The name of the newly created backup.

Return type

str

saveHiveTree(path=())

setAuthAdmin(name, isadmin)

setCellUser(iden)

Switch to another user (admin only).

This API allows remote admin/service accounts to impersonate a user. Used mostly by services that manage their own authentication/sessions.

setHiveKey(path, valu)

setRoleRules(iden, rules, gateiden=None)

setUserAdmin(iden, admin, gateiden=None)

setUserArchived(useriden, archived)

setUserEmail(useriden, email)

setUserLocked(useriden, locked)

async setUserPasswd(iden, passwd)

setUserProfInfo(iden, name, valu)

setUserRoles(useriden, roleidens)

setUserRules(iden, rules, gateiden=None)

trimNexsLog(consumers=None, timeout=60)

Rotate and cull the Nexus log (and those of any consumers) at the current offset.

Note

If the consumers argument is provided they will first be checked if online before rotating and raise otherwise. After rotation, all consumers must catch-up to the offset to cull at before executing the cull, and will raise otherwise.

Parameters

• consumers (list or None) – Optional list of telepath URLs for downstream Nexus log consumers.

• timeout (int) – Time in seconds to wait for downstream consumers to be caught up.

Returns

The offset that the Nexus log was culled up to and including.

Return type

int

tryUserPasswd(name, passwd)

waitNexsOffs(offs, timeout=None)

Wait for the Nexus log to write an offset.

Parameters

• offs (int) – The offset to wait for.

• timeout (int or None) – An optional timeout in seconds.

Returns

True if the offset was written, False if it timed out.

Return type

bool

synapse.lib.cell.SLAB_MAP_SIZE = 134217728

Base classes for the synapse “cell” microservice architecture.

synapse.lib.cell.adminapi(log=False)

Decorator for CellApi (and subclasses) for requiring a method to be called only by an admin user.

Parameters

log (bool) – If set to True, log the user, function and arguments.

synapse.lib.certdir module

class synapse.lib.certdir.CertDir(path=None)

Bases: object

Certificate loading/generation/signing utilities.

Features:

• Locates and load certificates, keys, and certificate signing requests (CSRs).

• Generates keypairs for users, hosts, and certificate authorities (CAs), supports both signed and self-signed.

• Generates certificate signing requests (CSRs) for users, hosts, and certificate authorities (CAs).

• Signs certificate signing requests (CSRs).

• Generates PKCS#12 archives for use in browser.

Parameters

path (str) – Optional path which can override the default path directory.

Notes

• All certificates will be loaded from and written to ~/.syn/certs by default. Set the environment variable SYN_CERT_DIR to override.

• All certificate generation methods create 4096 bit RSA keypairs.

• All certificate signing methods use sha256 as the signature algorithm.

• CertDir does not currently support signing CA CSRs.

addCertPath(*path)

delCertPath(*path)

genCaCert(name, signas=None, outp=None, save=True)

Generates a CA keypair.

Parameters

• name (str) – The name of the CA keypair.

• signas (str) – The CA keypair to sign the new CA with.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Make a CA named “myca”:

mycakey, mycacert = cdir.genCaCert(‘myca’)

Returns

Tuple containing the private key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genClientCert(name, outp=None)

Generates a user PKCS #12 archive. Please note that the resulting file will contain private key material.

Parameters

• name (str) – The name of the user keypair.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Make the PKC12 object for user “myuser”:

myuserpkcs12 = cdir.genClientCert(‘myuser’)

Returns

The PKCS #12 archive.

Return type

OpenSSL.crypto.PKCS12

genHostCert(name, signas=None, outp=None, csr=None, sans=None, save=True)

Generates a host keypair.

Parameters

• name (str) – The name of the host keypair.

• signas (str) – The CA keypair to sign the new host keypair with.

• outp (synapse.lib.output.Output) – The output buffer.

• csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

• sans (list) – List of subject alternative names.

Examples

Make a host keypair named “myhost”:

myhostkey, myhostcert = cdir.genHostCert(‘myhost’)

Returns

Tuple containing the private key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genHostCsr(name, outp=None)

Generates a host certificate signing request.

Parameters

• name (str) – The name of the host CSR.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the host key named “myhost”:

cdir.genHostCsr(‘myhost’)

Returns

The bytes of the CSR.

Return type

bytes

genUserCert(name, signas=None, outp=None, csr=None, save=True)

Generates a user keypair.

Parameters

• name (str) – The name of the user keypair.

• signas (str) – The CA keypair to sign the new user keypair with.

• outp (synapse.lib.output.Output) – The output buffer.

• csr (OpenSSL.crypto.PKey) – The CSR public key when generating the keypair from a CSR.

Examples

Generate a user cert for the user “myuser”:

myuserkey, myusercert = cdir.genUserCert(‘myuser’)

Returns

Tuple containing the key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

genUserCsr(name, outp=None)

Generates a user certificate signing request.

Parameters

• name (str) – The name of the user CSR.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

Generate a CSR for the user “myuser”:

cdir.genUserCsr(‘myuser’)

Returns

The bytes of the CSR.

Return type

bytes

getCaCert(name)

Loads the X509 object for a given CA.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the certificate for the CA “myca”

mycacert = cdir.getCaCert(‘myca’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getCaCertPath(name)

Gets the path to a CA certificate.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the path to the CA certificate for the CA “myca”:

mypath = cdir.getCACertPath(‘myca’)

Returns

The path if exists.

Return type

str

getCaCerts()

Return a list of CA certs from the CertDir.

Returns

List of CA certificates.

Return type

[OpenSSL.crypto.X509]

getCaKey(name)

Loads the PKey object for a given CA keypair.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the private key for the CA “myca”:

mycakey = cdir.getCaKey(‘myca’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getCaKeyPath(name)

Gets the path to a CA key.

Parameters

name (str) – The name of the CA keypair.

Examples

Get the path to the private key for the CA “myca”:

mypath = cdir.getCAKeyPath(‘myca’)

Returns

The path if exists.

Return type

str

getClientCert(name)

Loads the PKCS12 archive object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the PKCS12 object for the user “myuser”:

mypkcs12 = cdir.getClientCert(‘myuser’)

Notes

The PKCS12 archive will contain private key material if it was created with CertDir or the easycert tool

Returns

The PKCS12 archive, if exists.

Return type

OpenSSL.crypto.PKCS12

getClientCertPath(name)

Gets the path to a client certificate.

Parameters

name (str) – The name of the client keypair.

Examples

Get the path to the client certificate for “myuser”:

mypath = cdir.getClientCertPath(‘myuser’)

Returns

The path if exists.

Return type

str

getClientSSLContext(certname=None)

Returns an ssl.SSLContext appropriate for initiating a TLS session

Parameters

certname – If specified, use the user certificate with the matching name to authenticate to the remote service.

getHostCaPath(name)

Gets the path to the CA certificate that issued a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the CA cert which issue the cert for “myhost”:

mypath = cdir.getHostCaPath(‘myhost’)

Returns

The path if exists.

Return type

str

getHostCert(name)

Loads the X509 object for a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the certificate object for the host “myhost”:

myhostcert = cdir.getHostCert(‘myhost’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getHostCertPath(name)

Gets the path to a host certificate.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the host certificate for the host “myhost”:

mypath = cdir.getHostCertPath(‘myhost’)

Returns

The path if exists.

Return type

str

getHostKey(name)

Loads the PKey object for a given host keypair.

Parameters

name (str) – The name of the host keypair.

Examples

Get the private key object for the host “myhost”:

myhostkey = cdir.getHostKey(‘myhost’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getHostKeyPath(name)

Gets the path to a host key.

Parameters

name (str) – The name of the host keypair.

Examples

Get the path to the host key for the host “myhost”:

mypath = cdir.getHostKeyPath(‘myhost’)

Returns

The path if exists.

Return type

str

getServerSSLContext(hostname=None, caname=None)

Returns an ssl.SSLContext appropriate to listen on a socket

Parameters

• hostname – if None, the value from socket.gethostname is used to find the key in the servers directory. This name should match the not-suffixed part of two files ending in .key and .crt in the hosts subdirectory

• caname – If not None, the given name is used to locate a CA certificate used to validate client SSL certs.

getUserCaPath(name)

Gets the path to the CA certificate that issued a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path to the CA cert which issue the cert for “myuser”:

mypath = cdir.getUserCaPath(‘myuser’)

Returns

The path if exists.

Return type

str

getUserCert(name)

Loads the X509 object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the certificate object for the user “myuser”:

myusercert = cdir.getUserCert(‘myuser’)

Returns

The certificate, if exists.

Return type

OpenSSL.crypto.X509

getUserCertPath(name)

Gets the path to a user certificate.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path for the user cert for “myuser”:

mypath = cdir.getUserCertPath(‘myuser’)

Returns

The path if exists.

Return type

str

getUserForHost(user, host)

Gets the name of the first existing user cert for a given user and host.

Parameters

• user (str) – The name of the user.

• host (str) – The name of the host.

Examples

Get the name for the “myuser” user cert at “cool.vertex.link”:

usercertname = cdir.getUserForHost(‘myuser’, ‘cool.vertex.link’)

Returns

The cert name, if exists.

Return type

str

getUserKey(name)

Loads the PKey object for a given user keypair.

Parameters

name (str) – The name of the user keypair.

Examples

Get the key object for the user key for “myuser”:

myuserkey = cdir.getUserKey(‘myuser’)

Returns

The private key, if exists.

Return type

OpenSSL.crypto.PKey

getUserKeyPath(name)

Gets the path to a user key.

Parameters

name (str) – The name of the user keypair.

Examples

Get the path to the user key for “myuser”:

mypath = cdir.getUserKeyPath(‘myuser’)

Returns

The path if exists.

Return type

str

importFile(path, mode, outp=None)

Imports certs and keys into the Synapse cert directory

Parameters

• path (str) – The path of the file to be imported.

• mode (str) – The certdir subdirectory to import the file into.

Examples

Import CA certifciate ‘mycoolca.crt’ to the ‘cas’ directory.

certdir.importFile(‘mycoolca.crt’, ‘cas’)

Notes

importFile does not perform any validation on the files it imports.

Returns

None

isCaCert(name)

Checks if a CA certificate exists.

Parameters

name (str) – The name of the CA keypair.

Examples

Check if the CA certificate for “myca” exists:

exists = cdir.isCaCert(‘myca’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isClientCert(name)

Checks if a user client certificate (PKCS12) exists.

Parameters

name (str) – The name of the user keypair.

Examples

Check if the client certificate “myuser” exists:

exists = cdir.isClientCert(‘myuser’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isHostCert(name)

Checks if a host certificate exists.

Parameters

name (str) – The name of the host keypair.

Examples

Check if the host cert “myhost” exists:

exists = cdir.isUserCert(‘myhost’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

isUserCert(name)

Checks if a user certificate exists.

Parameters

name (str) – The name of the user keypair.

Examples

Check if the user cert “myuser” exists:

exists = cdir.isUserCert(‘myuser’)

Returns

True if the certificate is present, False otherwise.

Return type

bool

saveCertPem(cert, path)

Save a certificate in PEM format to a file outside the certdir.

savePkeyPem(pkey, path)

Save a private key in PEM format to a file outside the certdir.

selfSignCert(cert, pkey)

Self-sign a certificate.

Parameters

• cert (OpenSSL.crypto.X509) – The certificate to sign.

• pkey (OpenSSL.crypto.PKey) – The PKey with which to sign the certificate.

Examples

Sign a given certificate with a given private key:

cdir.selfSignCert(mycert, myotherprivatekey)

Returns

None

signCertAs(cert, signas)

Signs a certificate with a CA keypair.

Parameters

• cert (OpenSSL.crypto.X509) – The certificate to sign.

• signas (str) – The CA keypair name to sign the new keypair with.

Examples

Sign a certificate with the CA “myca”:

cdir.signCertAs(mycert, ‘myca’)

Returns

None

signHostCsr(xcsr, signas, outp=None, sans=None)

Signs a host CSR with a CA keypair.

Parameters

• cert (OpenSSL.crypto.X509Req) – The certificate signing request.

• signas (str) – The CA keypair name to sign the CSR with.

• outp (synapse.lib.output.Output) – The output buffer.

• sans (list) – List of subject alternative names.

Examples

Sign a host key with the CA “myca”:

cdir.signHostCsr(mycsr, ‘myca’)

Returns

Tuple containing the public key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

signUserCsr(xcsr, signas, outp=None)

Signs a user CSR with a CA keypair.

Parameters

• cert (OpenSSL.crypto.X509Req) – The certificate signing request.

• signas (str) – The CA keypair name to sign the CSR with.

• outp (synapse.lib.output.Output) – The output buffer.

Examples

cdir.signUserCsr(mycsr, ‘myca’)

Returns

Tuple containing the public key and certificate objects.

Return type

((OpenSSL.crypto.PKey, OpenSSL.crypto.X509))

valUserCert(byts, cacerts=None)

Validate the PEM encoded x509 user certificate bytes and return it.

Parameters

• byts (bytes) – The bytes for the User Certificate.

• cacerts (tuple) – A tuple of OpenSSL.crypto.X509 CA Certificates.

Raises

OpenSSL.crypto.X509StoreContextError – If the certificate is not valid.

Returns

The certificate, if it is valid.

Return type

OpenSSL.crypto.X509

synapse.lib.certdir.addCertPath(path)

synapse.lib.certdir.delCertPath(path)

synapse.lib.certdir.getCertDir() → synapse.lib.certdir.CertDir

Get the singleton CertDir instance.

Returns

A certdir object.

Return type

CertDir

synapse.lib.certdir.getCertDirn() → str

Get the expanded default path used by the singleton CertDir instance.

Returns

The path string.

Return type

str

synapse.lib.certdir.iterFqdnUp(fqdn)

synapse.lib.chop module

synapse.lib.chop.TagMatchRe = regex.Regex('([\\w*]+\\.)*[\\w*]+', flags=regex.V0)

Shared primitive routines for chopping up strings and values.

synapse.lib.chop.digits(text)

synapse.lib.chop.hexstr(text)

Ensure a string is valid hex.

Parameters

text (str) – String to normalize.

Examples

Norm a few strings:

hexstr(‘0xff00’) hexstr(‘ff00’)

Notes

Will accept strings prefixed by ‘0x’ or ‘0X’ and remove them.

Returns

Normalized hex string.

Return type

str

synapse.lib.chop.intstr(text)

synapse.lib.chop.onespace(text)

synapse.lib.chop.printables(text)

synapse.lib.chop.replaceUnicodeDashes(valu)

Replace unicode dashes in a string with regular dashes.

Parameters

valu (str) – A string.

Returns

A new string with replaced dashes.

Return type

str

synapse.lib.chop.stormstring(s)

Make a string storm safe by escaping backslashes and double quotes.

Parameters

s (str) – String to make storm safe.

Notes

This does not encapsulate a string in double quotes.

Returns

A string which can be embedded directly into a storm query.

Return type

str

synapse.lib.chop.tag(text)

synapse.lib.chop.tagpath(text)

synapse.lib.chop.tags(norm)

Divide a normalized tag string into hierarchical layers.

synapse.lib.chop.validateTagMatch(tag)

Raises an exception if tag is not a valid tagmatch (i.e. a tag that might have globs)

synapse.lib.cli module

class synapse.lib.cli.Cli

Bases: synapse.lib.base.Base

A modular / event-driven CLI base object.

addCmdClass(ctor, **opts)

Add a Cmd subclass to this cli.

async addSignalHandlers()

Register SIGINT signal handler with the ioloop to cancel the currently running cmdloop task.

get(name, defval=None)

getCmdByName(name)

Return a Cmd instance by name.

getCmdNames()

Return a list of all the known command names for the CLI.

getCmdPrompt()

Get the command prompt.

Returns

Configured command prompt

Return type

str

histfile = 'cmdr_history'

initCmdClasses()

printf(mesg, addnl=True, color=None)

async prompt(text=None)

Prompt for user input from stdin.

async runCmdLine(line)

Run a single command line.

Parameters

line (str) – Line to execute.

Examples

Execute the ‘woot’ command with the ‘help’ switch:

await cli.runCmdLine(‘woot –help’)

Returns

Arbitrary data from the cmd class.

Return type

object

async runCmdLoop()

Run commands from a user in an interactive fashion until fini() or EOFError is raised.

set(name, valu)

class synapse.lib.cli.Cmd(cli, **opts)

Bases: object

Base class for modular commands in the synapse CLI.

getCmdBrief()

Return the single-line description for this command.

getCmdDoc()

Return the help/doc output for this command.

getCmdItem()

Get a reference to the object we are commanding.

getCmdName()

getCmdOpts(text)

Use the _cmd_syntax def to split/parse/normalize the cmd line.

Parameters

text (str) – Command to process.

Notes

This is implemented independent of argparse (et al) due to the need for syntax aware argument splitting. Also, allows different split per command type

Returns

An opts dictionary.

Return type

dict

printf(mesg, addnl=True, color=None)

async runCmdLine(line)

Run a line of command input for this command.

Parameters

line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdHelp(cli, **opts)

Bases: synapse.lib.cli.Cmd

List commands and display help output.

Example

help foocmd

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdLocals(cli, **opts)

Bases: synapse.lib.cli.Cmd

List the current locals for a given CLI object.

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.cli.CmdQuit(cli, **opts)

Bases: synapse.lib.cli.Cmd

Quit the current command line interpreter.

Example

quit

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

synapse.lib.cmd module

class synapse.lib.cmd.Parser(prog=None, outp=<synapse.lib.output.OutPut object>, **kwargs)

Bases: argparse.ArgumentParser

exit(status=0, message=None)

Argparse expects exit() to be a terminal function and not return. As such, this function must raise an exception instead.

synapse.lib.cmdr module

async synapse.lib.cmdr.getItemCmdr(cell, outp=None, color=False, **opts)

Construct and return a cmdr for the given remote cell.

Parameters

• cell – Cell proxy being commanded.

• outp – Output helper object.

• color (bool) – If true, enable colorized output.

• **opts – Additional options pushed into the Cmdr locs.

Examples

Get the cmdr for a proxy:

cmdr = await getItemCmdr(foo)

Returns

A Cli instance with Cmds loaeded into it.

Return type

s_cli.Cli

async synapse.lib.cmdr.runItemCmdr(item, outp=None, color=False, **opts)

Create a cmdr for the given item and run the cmd loop.

Parameters

• item – Cell proxy being commanded.

• outp – Output helper object.

• color (bool) – If true, enable colorized output.

• **opts – Additional options pushed into the Cmdr locs.

Notes

This function does not return while the command loop is run.

Examples

Run the Cmdr for a proxy:

await runItemCmdr(foo)

Returns

This function returns None.

Return type

None

synapse.lib.config module

class synapse.lib.config.Config(schema, conf=None, envar_prefix=None)

Bases: collections.abc.MutableMapping

Synapse configuration helper based on JSON Schema.

Parameters

• schema (dict) – The JSON Schema (draft v7) which to validate configuration data against.

• conf (dict) – Optional, a set of configuration data to preload.

• envar_prefix (str) – Optional, a prefix used when collecting configuration data from environment variables.

Notes

This class implements the collections.abc.MutableMapping class, so it may be used where a dictionary would otherwise be used.

The default values provided in the schema must be able to be recreated from the repr() of their Python value.

Default values are not loaded into the configuration data until the reqConfValid() method is called.

asDict()

Get a copy of configuration data.

Returns

A copy of the configuration data.

Return type

dict

getArgParseArgs()

getCmdlineMapping()

classmethod getConfFromCell(cell, conf=None, envar_prefix=None)

Get a Config object from a Cell directly (either the ctor or the instance thereof).

Returns

A Config object.

Return type

Config

getEnvarMapping()

Get a mapping of config values to envars.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

reqConfValid()

Validate that the loaded configuration data is valid according to the schema.

Notes

The validation set does set any default values which are not currently set for configuration options.

Returns

This returns nothing.

Return type

None

reqConfValu(key)

Get a configuration value. If that value is not present in the schema or is not set, then raise an exception.

Parameters

key (str) – The key to require.

Returns

The requested value.

setConfFromEnvs()

Set configuration options from environment variables.

Notes

Environment variables are resolved from configuration options after doing the following transform:

• Replace : characters with _.

• Add a config provided prefix, if set.

• Uppercase the string.

• Resolve the environment variable

• If the environment variable is set, set the config value to the results of yaml.yaml_safeload() on the value.

Configuration values which have the hideconf value set to True are not resolved from environment variables.

Examples

For the configuration value auth:passwd, the environment variable is resolved as AUTH_PASSWD. With the prefix cortex, the the environment variable is resolved as CORTEX_AUTH_PASSWD.

Returns

Returns None.

Return type

None

setConfFromOpts(opts)

Set the opts for a conf object from a namespace object.

Parameters

• opts (argparse.Namespace) – A Namespace object made from parsing args with an ArgumentParser

• getArgumentParser. (made with) –

Returns

Returns None.

Return type

None

synapse.lib.config.getJsSchema(confbase, confdefs)

Generate a Synapse JSON Schema for a Cell using a pair of confbase and confdef values.

Parameters

• confbase (dict) – A JSON Schema dictionary of properties for the object. This content has precedence over the confdefs argument.

• confdefs (dict) – A JSON Schema dictionary of properties for the object.

Notes

This generated a JSON Schema draft 7 schema for a single object, which does not allow for additional properties to be set on it. The data in confdefs is implementer controlled and is welcome to specify

Returns

A complete JSON schema.

Return type

dict

synapse.lib.config.getJsValidator(schema, use_default=True)

Get a fastjsonschema callable.

Parameters

• schema (dict) – A JSON Schema object.

• use_default (bool) – Whether to insert “default” key arguments into the validated data structure.

Returns

A callable function that can be used to validate data against the json schema.

Return type

callable

synapse.lib.config.make_envar_name(key, prefix=None)

Convert a colon delimited string into an uppercase, underscore delimited string.

Parameters

• key (str) – Config key to convert.

• prefix (str) – Optional string prefix to prepend the the config key.

Returns

The string to lookup against a envar.

Return type

str

synapse.lib.const module

synapse.lib.coro module

Async/Coroutine related utilities.

class synapse.lib.coro.Event(*, loop=None)

Bases: asyncio.locks.Event

async timewait(timeout=None)

class synapse.lib.coro.GenrHelp(genr)

Bases: object

async list()

async spin()

async synapse.lib.coro.agen(item)

Wrap an async_generator or generator in an async_generator.

Notes

Do not use this for a synchronous generator which would cause non-blocking IO; otherwise that IO will block the ioloop.

async synapse.lib.coro.event_wait(event: asyncio.locks.Event, timeout=None)

Wait on an an asyncio event with an optional timeout

Returns

true if the event got set, False if timed out

synapse.lib.coro.executor(func, *args, **kwargs)

Execute a non-coroutine function in the ioloop executor pool.

Parameters

• func – Function to execute.

• *args – Args for the function.

• **kwargs – Kwargs for the function.

Examples

Execute a blocking API call in the executor pool:

import requests

def block(url, params=None):
    return requests.get(url, params=params).json()

fut = s_coro.executor(block, 'http://some.tld/thign')
resp = await fut

Returns

An asyncio future.

Return type

asyncio.Future

async synapse.lib.coro.forked(func, *args, **kwargs)

Execute a target function in the forked process pool.

Parameters

• func – The target function.

• *args – Function positional arguments.

• **kwargs – Function keyword arguments.

Returns

The target function return.

Raises

• The function may raise from the target function, or raise a s_exc.FatalErr in the event of a broken forked –

• process pool. The fatalerr represents a unrecoverable application state. –

synapse.lib.coro.genrhelp(f)

synapse.lib.coro.iscoro(item)

async synapse.lib.coro.ornot(func, *args, **kwargs)

Calls func and awaits it if a returns a coroutine.

Note

This is useful for implementing a function that might take a telepath proxy object or a local object, and you must call a non-async method on that object.

This is also useful when calling a callback that might either be a coroutine function or a regular function.

Usage:

ok = await s_coro.ornot(maybeproxy.allowed, ‘path’)

synapse.lib.coro.set_pool_logging(logger_, logconf)

async synapse.lib.coro.spawn(todo, timeout=None, ctx=None)

Run a todo (func, args, kwargs) tuple in a multiprocessing subprocess.

Parameters

• todo (tuple) – A tuple of function, *args, and **kwargs.

• timeout (int) – The timeout to wait for the todo function to finish.

• ctx (multiprocess.Context) – A optional multiprocessing context object.

Notes

The contents of the todo tuple must be able to be pickled for execution. This means that locally bound functions are not eligible targets for spawn.

Returns

The return value of executing the todo function.

async synapse.lib.coro.waittask(task, timeout=None)

Await a task without cancelling it when you time out.

Returns

True if the task completed before the timeout.

Return type

boolean

synapse.lib.datfile module

Utilities for handling data files embedded within python packages.

synapse.lib.datfile.openDatFile(datpath)

Open a file-like object using a pkg relative path.

Example

fd = openDatFile(‘foopkg.barpkg/wootwoot.bin’)

synapse.lib.dyndeps module

synapse.lib.dyndeps.getDynLocal(name)

Dynamically import a python module and return a local.

Example

cls = getDynLocal(‘foopkg.barmod.BlahClass’) blah = cls()

synapse.lib.dyndeps.getDynMeth(name)

Retrieve and return an unbound method by python path.

synapse.lib.dyndeps.getDynMod(name)

Dynamically import a python module and return a ref (or None).

Example

mod = getDynMod(‘foo.bar’)

synapse.lib.dyndeps.runDynTask(task)

Run a dynamic task and return the result.

Example

foo = runDynTask( (‘baz.faz.Foo’, (), {} ) )

synapse.lib.dyndeps.tryDynFunc(name, *args, **kwargs)

Dynamically import a module and call a function or raise an exception.

synapse.lib.dyndeps.tryDynLocal(name)

Dynamically import a module and return a module local or raise an exception.

synapse.lib.dyndeps.tryDynMod(name)

Dynamically import a python module or exception.

synapse.lib.encoding module

synapse.lib.encoding.addFormat(name, fn, opts)

Add an additional ingest file format

synapse.lib.encoding.decode(name, byts, **opts)

Decode the given byts with the named decoder. If name is a comma separated list of decoders, loop through and do them all.

Example

byts = s_encoding.decode(‘base64’,byts)

Note: Decoder names may also be prefixed with +

to encode for that name/layer.

synapse.lib.encoding.encode(name, item, **opts)

synapse.lib.encoding.iterdata(fd, close_fd=True, **opts)

Iterate through the data provided by a file like object.

Optional parameters may be used to control how the data is deserialized.

Examples

The following example show use of the iterdata function.:

with open('foo.csv','rb') as fd:
    for row in iterdata(fd, format='csv', encoding='utf8'):
        dostuff(row)

Parameters

• fd (file) – File like object to iterate over.

• close_fd (bool) – Default behavior is to close the fd object. If this is not true, the fd will not be closed.

• **opts (dict) – Ingest open directive. Causes the data in the fd to be parsed according to the ‘format’ key and any additional arguments.

Yields

An item to process. The type of the item is dependent on the format parameters.

synapse.lib.gis module

synapse.lib.gis.bbox(lat, lon, dist)

Calculate a min/max bounding box for the circle defined by lalo/dist.

Parameters

• lat (float) – The latitude in degrees

• lon (float) – The longitude in degrees

• dist (int) – A distance in geo:dist base units (mm)

Returns

(latmin, latmax, lonmin, lonmax)

Return type

(float,float,float,float)

synapse.lib.gis.dms2dec(degs, mins, secs)

Convert degrees, minutes, seconds lat/long form to degrees float.

Parameters

• degs (int) – Degrees

• mins (int) – Minutes

• secs (int) – Seconds

Returns

Degrees

Return type

(float)

synapse.lib.gis.haversine(px, py, r=6371008800.0)

Calculate the haversine distance between two points defined by (lat,lon) tuples.

Parameters

• px ((float,float)) – lat/long position 1

• py ((float,float)) – lat/long position 2

• r (float) – Radius of sphere

Returns

Distance in mm.

Return type

(int)

synapse.lib.gis.latlong(text)

Chop a latlong string and return (float,float). Does not perform validation on the coordinates.

Parameters

text (str) – A longitude,latitude string.

Returns

A longitude, latitude float tuple.

Return type

(float,float)

synapse.lib.gis.near(point, dist, points)

Determine if the given point is within dist of any of points.

Parameters

• point ((float,float)) – A latitude, longitude float tuple.

• dist (int) – A distance in mm ( base units )

• points (list) – A list of latitude, longitude float tuples to compare against.

synapse.lib.grammar module

synapse.lib.grammar.isBasePropNoPivprop(name)

synapse.lib.grammar.isCmdName(name)

synapse.lib.grammar.isFormName(name)

synapse.lib.grammar.isPropName(name)

synapse.lib.grammar.isUnivName(name)

synapse.lib.grammar.meh(txt, off, cset)

synapse.lib.grammar.nom(txt, off, cset, trim=True)

Consume chars in set from the string and return (subtxt,offset).

Example

text = “foo(bar)” chars = set(‘abcdefghijklmnopqrstuvwxyz’)

name,off = nom(text,0,chars)

Note:

This really shouldn’t be used for new code

synapse.lib.grammar.parse_float(text, off)

synapse.lib.hashitem module

synapse.lib.hashitem.hashitem(item)

Generate a uniq hash for the JSON compatible primitive data structure.

synapse.lib.hashitem.normdict(item)

synapse.lib.hashitem.normitem(item)

synapse.lib.hashitem.normiter(item)

synapse.lib.hashset module

class synapse.lib.hashset.HashSet

Bases: object

digests()

Get a list of (name, bytes) tuples for the hashes in the hashset.

eatfd(fd)

Consume all the bytes from a file like object.

Example

hset = HashSet() hset.eatfd(fd)

guid()

Use elements from this hash set to create a unique (re)identifier.

update(byts)

Update all the hashes in the set with the given bytes.

synapse.lib.health module

class synapse.lib.health.HealthCheck(iden)

Bases: object

getStatus()

pack()

setStatus(valu)

update(name, status, mesg='', data=None)

Append a new component to the Healcheck object.

Parameters

• name (str) – Name of the reported component.

• status (str) – nomdinal/degraded/failed status code.

• mesg (str) – Optional message about the component status.

• data (dict) – Optional arbitrary dictionary of additional metadata about the component.

Returns

None

synapse.lib.hive module

class synapse.lib.hive.Hive

Bases: synapse.lib.nexus.Pusher, synapse.telepath.Aware

An optionally persistent atomically accessed tree which implements primitives for use in making distributed/clustered services.

async add(full, valu)

Atomically increments a node’s value.

async dict(full, nexs=False)

Open a HiveDict at the given full path.

Parameters

full (tuple) – A full path tuple.

Returns

A HiveDict for the full path.

Return type

HiveDict

dir(full)

List subnodes of the given Hive path.

Parameters

full (tuple) – A full path tuple.

Notes

This returns None if there is not a node at the path.

Returns

A list of tuples. Each tuple contains the name, node value, and the number of children nodes.

Return type

list

async exists(full)

Returns whether the Hive path has already been created.

async get(full, defv=None)

Get the value of a node at a given path.

Parameters

full (tuple) – A full path tuple.

Returns

Arbitrary node value.

async getHiveAuth()

Retrieve a HiveAuth for hive standalone or non-cell uses.

Note

This is for the hive’s own auth, or for non-cell auth. It isn’t the same auth as for a cell

async getTeleApi(link, mesg, path)

Return a shared object for this link. :param link: A network link. :type link: synapse.lib.link.Link :param mesg: The tele:syn handshake message. :type mesg: (str,dict)

async loadHiveTree(tree, path=(), trim=False)

async open(full)

Open and return a hive Node().

Parameters

full (tuple) – A full path tuple.

Returns

A Hive node.

Return type

Node

async pop(full, nexs=False)

Remove and return the value for the given node.

async rename(oldpath, newpath)

Moves a node at oldpath and all its descendant nodes to newpath. newpath must not exist

async saveHiveTree(path=())

async set(full, valu, nexs=False)

A set operation at the hive level (full path).

async storNodeDele(path)

async storNodeValu(full, valu)

class synapse.lib.hive.HiveApi

Bases: synapse.lib.base.Base

async addAndSync(path, valu, iden)

async edits()

async get(full)

async loadHiveTree(tree, path=(), trim=False)

async popAndSync(path, iden, nexs=False)

async saveHiveTree(path=())

async setAndSync(path, valu, iden, nexs=False)

async treeAndSync(path, iden)

class synapse.lib.hive.HiveDict

Bases: synapse.lib.base.Base

get(name, default=None)

items()

pack()

async pop(name, default=None)

async set(name, valu)

setdefault(name, valu)

values()

class synapse.lib.hive.Node

Bases: synapse.lib.base.Base

A single node within the Hive tree.

async add(valu)

Increments existing node valu

async dict(nexs=False)

Get a HiveDict for this Node.

Returns

A HiveDict for this Node.

Return type

HiveDict

dir()

get(name)

name()

async open(path)

Open a child Node of the this Node.

Parameters

path (tuple) – A child path of the current node.

Returns

A Node at the child path.

Return type

Node

parent()

async pop(path=())

async set(valu)

class synapse.lib.hive.SlabHive

Bases: synapse.lib.hive.Hive

async storNodeDele(full)

async storNodeValu(full, valu)

class synapse.lib.hive.TeleHive

Bases: synapse.lib.hive.Hive

A Hive that acts as a consistent read cache for a telepath proxy Hive

async add(path, valu)

Atomically increments a node’s value.

async get(path)

Get the value of a node at a given path.

Parameters

full (tuple) – A full path tuple.

Returns

Arbitrary node value.

async open(path)

Open and return a hive Node().

Parameters

full (tuple) – A full path tuple.

Returns

A Hive node.

Return type

Node

async pop(path, nexs=False)

Remove and return the value for the given node.

async set(path, valu, nexs=False)

A set operation at the hive level (full path).

synapse.lib.hive.iterpath(path)

async synapse.lib.hive.opendir(dirn, conf=None)

async synapse.lib.hive.openurl(url, **opts)

synapse.lib.hiveauth module

class synapse.lib.hiveauth.Auth

Bases: synapse.lib.nexus.Pusher

Auth is a user authentication and authorization stored in a Hive. Users correspond to separate logins with different passwords and potentially different privileges.

Users are assigned “rules”. These rules are evaluated in order until a rule matches. Each rule is a tuple of boolean, and a rule path (a sequence of strings). Rules that are prefixes of a privilege match, i.e. a rule (‘foo’,) will match (‘foo’, ‘bar’).

Roles are just collections of rules. When a user is “granted” a role those rules are assigned to that user. Unlike in an RBAC system, users don’t explicitly assume a role; they are merely a convenience mechanism to easily assign the same rules to multiple users.

Authgates are objects that manage their own authorization. Each AuthGate has roles and users subkeys which contain rules specific to that user or role for that AuthGate. The roles and users of an AuthGate, called GateRole and GateUser respectively, contain the iden of a role or user defined prior and rules specific to that role or user; they do not duplicate the metadata of the role or user.

Node layout:

Auth root (passed into constructor)
├ roles
│   ├ <role iden 1>
│   ├ ...
│   └ last role
├ users
│   ├ <user iden 1>
│   ├ ...
│   └ last user
└ authgates
    ├ <iden 1>
    │   ├ roles
    │   │   ├ <role iden 1>
    │   │   ├ ...
    │   │   └ last role
    │   └ users
    │       ├ <user iden 1>
    │       ├ ...
    │       └ last user
    ├ <iden 2>
    │   ├ ...
    └ ... last authgate

async addAuthGate(iden, authgatetype)

Retrieve AuthGate by iden. Create if not present.

Note

Not change distributed

Returns

(HiveAuthGate)

async addRole(name, iden=None)

async addUser(name, passwd=None, email=None, iden=None)

Add a User to the Hive.

Parameters

• name (str) – The name of the User.

• passwd (str) – A optional password for the user.

• email (str) – A optional email for the user.

• iden (str) – A optional iden to use as the user iden.

Returns

A Hive User.

Return type

HiveUser

async delAuthGate(iden)

Delete AuthGate by iden.

Note

Not change distributed

async delRole(iden)

async delUser(iden)

getAuthGate(iden)

getAuthGates()

async getRoleByName(name)

async getUserByName(name)

Get a user by their username.

Parameters

name (str) – Name of the user to get.

Returns

A Hive User. May return None if there is no user by the requested name.

Return type

HiveUser

async getUserIdenByName(name)

reqAuthGate(iden)

async reqRole(iden)

async reqRoleByName(name)

async reqUser(iden)

async reqUserByName(name)

async reqUserByNameOrIden(name)

role(iden)

roles()

async setRoleInfo(iden, name, valu, gateiden=None)

async setRoleName(iden, name)

async setUserInfo(iden, name, valu, gateiden=None)

async setUserName(iden, name)

user(iden)

users()

class synapse.lib.hiveauth.AuthGate

Bases: synapse.lib.base.Base

The storage object for object specific rules for users/roles.

async delete()

async genRoleInfo(iden)

async genUserInfo(iden)

pack()

class synapse.lib.hiveauth.HiveRole

Bases: synapse.lib.hiveauth.HiveRuler

A role within the Hive authorization subsystem.

A role in HiveAuth exists to bundle rules together so that the same set of rules can be applied to multiple users.

clearAuthCache()

async genGateInfo(gateiden)

pack()

async setName(name)

class synapse.lib.hiveauth.HiveRuler

Bases: synapse.lib.base.Base

A HiveNode that holds a list of rules. This includes HiveUsers, HiveRoles, and the AuthGate variants of those

async addRule(rule, indx=None, gateiden=None, nexs=True)

async delRule(rule, gateiden=None)

getRules(gateiden=None)

async setRules(rules, gateiden=None, nexs=True)

class synapse.lib.hiveauth.HiveUser

Bases: synapse.lib.hiveauth.HiveRuler

A user (could be human or computer) of the system within HiveAuth.

Cortex-wide rules are stored here. AuthGate-specific rules for this user are stored in an GateUser.

allowed(perm, default=None, gateiden=None)

clearAuthCache()

confirm(perm, default=None, gateiden=None)

async genGateInfo(gateiden)

getRoles()

async grant(roleiden, indx=None)

hasRole(iden)

isAdmin(gateiden=None)

isLocked()

pack(packroles=False)

raisePermDeny(perm, gateiden=None)

async revoke(iden, nexs=True)

async setAdmin(admin, gateiden=None, logged=True)

async setArchived(archived)

async setLocked(locked, logged=True)

async setName(name)

async setPasswd(passwd, nexs=True)

async setRoles(roleidens)

Replace all the roles for a given user with a new list of roles.

Parameters

roleidens (list) – A list of roleidens.

Notes

The roleiden for the “all” role must be present in the new list of roles. This replaces all existing roles that the user has with the new roles.

Returns

None

async tryPasswd(passwd)

synapse.lib.hiveauth.getShadow(passwd)

synapse.lib.httpapi module

class synapse.lib.httpapi.ActiveV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.AuthAddRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.AuthAddUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.AuthDelRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.AuthGrantV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

async get()

async post()

class synapse.lib.httpapi.AuthRevokeV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/grant?user=iden&role=iden

async get()

async post()

class synapse.lib.httpapi.AuthRoleV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get(iden)

async post(iden)

class synapse.lib.httpapi.AuthRolesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.AuthUserPasswdV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post(iden)

class synapse.lib.httpapi.AuthUserV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get(iden)

async post(iden)

class synapse.lib.httpapi.AuthUsersV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.CoreInfoV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/core/info

async get()

class synapse.lib.httpapi.FeedV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/feed

Examples

Example data:

{
    'name': 'syn.nodes',
    'view': null,
    'items': [...],
}

async post()

class synapse.lib.httpapi.Handler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

on_connection_close()

Called in async handlers if the client closed the connection.

Override this to clean up resources associated with long-lived connections. Note that this method is called only if the connection was closed during asynchronous processing; if you need to do cleanup after every request override on_finish instead.

Proxies may keep a connection open for a time (perhaps indefinitely) after the client has gone away, so this method may not be called promptly after the end user closes their connection.

prepare()

Called at the beginning of a request before get/post/etc.

Override this method to perform common initialization regardless of the request method.

Asynchronous support: Use async def or decorate this method with .gen.coroutine to make it asynchronous. If this method returns an Awaitable execution will not proceed until the Awaitable is done.

New in version 3.1: Asynchronous support.

class synapse.lib.httpapi.HandlerBase

Bases: object

async allowed(perm, gateiden=None)

Return true if there is a logged in user with the given permissions.

NOTE: This API sets up HTTP response values if it returns False.

NOTE: This API uses the Handler.getAuthCell() abstraction and is safe for use

in split-auth cells.

async authenticated()

check_origin(origin)

getAuthCell()

Return a reference to the cell used for auth operations.

getCustomHeaders()

getJsonBody(validator=None)

async getUserBody()

Helper function to confirm that there is a auth user and a valid JSON body in the request.

Returns

The user and body of the request as deserialized JSON, or a tuple of s_common.novalu

objects if there was no user or json body.

Return type

(User, object)

initialize(cell)

isOrigHost(origin)

loadJsonMesg(byts, validator=None)

options()

async reqAuthAdmin()

async reqAuthUser()

sendAuthRequired()

sendRestErr(code, mesg)

sendRestExc(e)

sendRestRetn(valu)

async sess(gen=True)

set_default_headers()

async user()

async useriden()

Return the user iden of the current session user.

NOTE: APIs should migrate toward using this rather than the heavy

Handler.user() API to facilitate reuse of handler objects with telepath references.

class synapse.lib.httpapi.HealthCheckV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.LoginV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.ModelNormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

async post()

class synapse.lib.httpapi.ModelV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.OnePassIssueV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

/api/v1/auth/onepass/issue

async post()

class synapse.lib.httpapi.ReqValidStormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.RobotHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.HandlerBase, tornado.web.RequestHandler

async get()

class synapse.lib.httpapi.Sess

Bases: synapse.lib.base.Base

addWebSock(sock)

delWebSock(sock)

async login(user)

async logout()

class synapse.lib.httpapi.StormCallV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.StormExportV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.StormHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

getCore()

class synapse.lib.httpapi.StormNodesV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.StormV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.StormHandler

async get()

async post()

class synapse.lib.httpapi.StormVarsGetV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async get()

class synapse.lib.httpapi.StormVarsPopV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.StormVarsSetV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

async post()

class synapse.lib.httpapi.StreamHandler(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.Handler

Subclass for Tornado streaming uploads.

Notes

• Async method prepare() is called after headers are read but before body processing.

• Sync method on_finish() can be used to cleanup after a request.

• Sync method on_connection_close() can be used to cleanup after a client disconnect.

• Async methods post(), put(), etc are called after the streaming has completed.

async data_received(chunk)

Implement this method to handle streamed request data.

Requires the .stream_request_body decorator.

May be a coroutine for flow control.

class synapse.lib.httpapi.WatchSockV1(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.WebSocket

A web-socket based API endpoint for distributing cortex events.

async onWatchMesg(byts)

async on_message(byts)

Handle incoming messages on the WebSocket

This method must be overridden.

Changed in version 4.5: on_message can be a coroutine.

class synapse.lib.httpapi.WebSocket(application: tornado.web.Application, request: tornado.httputil.HTTPServerRequest, **kwargs: Any)

Bases: synapse.lib.httpapi.HandlerBase, tornado.websocket.WebSocketHandler

async xmit(name, **info)

synapse.lib.ingest module

synapse.lib.interval module

A few utilities for dealing with intervals.

synapse.lib.interval.fold(*vals)

Initialize a new (min,max) tuple interval from values.

Parameters

*vals ([int,...]) – A list of values (or Nones)

Returns

A (min,max) interval tuple or None

Return type

((int,int))

synapse.lib.interval.overlap(ival0, ival1)

Determine if two interval tuples have overlap.

Parameters

• iv0 ((int,int)) – An interval tuple

• iv1 ((int,int)) –

Returns

True if the intervals overlap, otherwise False

Return type

(bool)

synapse.lib.interval.parsetime(text)

Parse an interval time string and return a (min,max) tuple.

Parameters

text (str) – A time interval string

Returns

A epoch millis epoch time string

Return type

((int,int))

synapse.lib.jsonstor module

class synapse.lib.jsonstor.JsonStor

Bases: synapse.lib.base.Base

A filesystem like storage mechanism that allows hirarchical lookup of reference counted “objects” that have individually editable properties.

#TODO json validation by path glob matches? (persists?) #TODO GUID ACCESS with index generation by type #TODO registered types jsonschema with optional write-back validation

async cmpDelPathObjProp(path, prop, valu)

async copyPathObj(oldp, newp)

async copyPathObjs(paths)

async delPathObj(path)

Remove a path and decref the object it references.

async delPathObjProp(path, prop)

async getPathList(path)

async getPathObj(path)

async getPathObjProp(path, prop)

async getPathObjs(path)

async hasPathObj(path)

async popPathObjProp(path, prop, defv=None)

async setPathLink(srcpath, dstpath)

Add a link from the given srcpath to the dstpath. NOTE: This causes the item at dstpath to be incref’d

async setPathObj(path, item)

Set (and/or reinitialize) the object at the given path.

NOTE: This will break any links by creating a new object.

async setPathObjProp(path, prop, valu)

class synapse.lib.jsonstor.JsonStorApi

Bases: synapse.lib.cell.CellApi

async addQueue(name, info)

addUserNotif(useriden, mesgtype, mesgdata=None)

async cmpDelPathObjProp(path, name, valu)

async copyPathObj(oldp, newp)

async copyPathObjs(paths)

async cullQueue(name, offs)

async delPathObj(path)

async delPathObjProp(path, name)

async delQueue(name)

delUserNotif(indx)

async getPathList(path)

async getPathObj(path)

async getPathObjProp(path, prop)

async getPathObjs(path)

getUserNotif(indx)

async getsQueue(name, offs, size=None, cull=True, wait=True)

async hasPathObj(path)

iterUserNotifs(useriden, size=None)

async popPathObjProp(path, prop)

async putsQueue(name, items)

async setPathLink(srcpath, dstpath)

async setPathObj(path, item)

async setPathObjProp(path, prop, valu)

watchAllUserNotifs(offs=None)

class synapse.lib.jsonstor.JsonStorCell

Bases: synapse.lib.cell.Cell

async addQueue(name, info)

async addUserNotif(useriden, mesgtype, mesgdata=None)

cellapi

alias of synapse.lib.jsonstor.JsonStorApi

async cmpDelPathObjProp(path, name, valu)

async copyPathObj(oldp, newp)

async copyPathObjs(paths)

async cullQueue(name, offs)

async delPathObj(path)

async delPathObjProp(path, name)

async delQueue(name)

async delUserNotif(indx)

async getPathList(path)

async getPathObj(path)

async getPathObjProp(path, prop)

async getPathObjs(path)

async getUserNotif(indx)

async getsQueue(name, offs, size=None, cull=True, wait=True)

async hasPathObj(path)

async initServiceStorage()

async iterUserNotifs(useriden, size=None)

async popPathObjProp(path, prop)

async putsQueue(name, items, reqid=None)

async setPathLink(srcpath, dstpath)

async setPathObj(path, item)

async setPathObjProp(path, prop, valu)

async watchAllUserNotifs(offs=None)

synapse.lib.jupyter module

class synapse.lib.jupyter.CmdrCore

Bases: synapse.lib.base.Base

A helper for jupyter/storm CLI interaction

async addFeedData(name, items, *, viewiden=None)

Add feed data to the cortex.

async eval(text, opts=None, num=None, cmdr=False)

A helper for executing a storm command and getting a list of packed nodes.

Parameters

• text (str) – Storm command to execute.

• opts (dict) – Opt to pass to the cortex during execution.

• num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.

• cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

Notes

The opts dictionary will not be used if cmdr=True.

Returns

A list of packed nodes.

Return type

list

async runCmdLine(text)

Run a line of text directly via cmdr.

async storm(text, opts=None, num=None, cmdr=False, suppress_logging=False)

A helper for executing a storm command and getting a list of storm messages.

Parameters

• text (str) – Storm command to execute.

• opts (dict) – Opt to pass to the cortex during execution.

• num (int) – Number of nodes to expect in the output query. Checks that with an assert statement.

• cmdr (bool) – If True, executes the line via the Cmdr CLI and will send output to outp.

• suppress_logging (bool) – If True, suppresses some logging related to Storm runtime exceptions.

Notes

The opts dictionary will not be used if cmdr=True.

Returns

A list of storm messages.

Return type

list

suppress_logging(suppress)

Context manager to suppress specific loggers.

synapse.lib.jupyter.genTempCoreProxy(mods=None)

Get a temporary cortex proxy.

synapse.lib.jupyter.genTempStormsvcProxy(cmdrcore, svcname, svcctor, conf=None)

synapse.lib.jupyter.getDocData(fp, root=None)

Parameters

• fn (str) – Name of the file to retrieve the data of.

• root (str) – Optional root path to look for a docdata directory in.

Notes

Will detect json/jsonl/yaml/mpk extensions and automatically decode that data if found; otherwise it returns bytes.

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns

May be deserialized data or bytes.

Return type

data

Raises

ValueError if the file does not exist or directory traversal attempted.. –

synapse.lib.jupyter.getDocPath(fn, root=None)

Helper for getting a documentation data file paths.

Parameters

• fn (str) – Name of the file to retrieve the full path for.

• root (str) – Optional root path to look for a docdata in.

Notes

Defaults to looking for the docdata directory in the current working directory. This behavior works fine for notebooks nested in the docs directory of synapse; but this root directory that is looked for may be overridden by providing an alternative root.

Returns

A file path.

Return type

str

Raises

ValueError if the file does not exist or directory traversal attempted.. –

async synapse.lib.jupyter.getItemCmdr(prox, outp=None, locs=None)

Get a Cmdr instance with prepopulated locs

async synapse.lib.jupyter.getTempCoreCmdr(mods=None, outp=None)

Get a CmdrCore instance which is backed by a temporary Cortex.

Parameters

• mods (list) – A list of additional CoreModules to load in the Cortex.

• outp – A output helper. Will be used for the Cmdr instance.

Notes

The CmdrCore returned by this should be fini()’d to tear down the temporary Cortex.

Returns

A CmdrCore instance.

Return type

CmdrCore

async synapse.lib.jupyter.getTempCoreCmdrStormsvc(svcname, svcctor, svcconf=None, outp=None)

Get a proxy to a Storm service and a CmdrCore instance backed by a temporary Cortex with the service added.

Parameters

• svcname (str) – Storm service name

• svcctor – Storm service constructor (e.g. Example.anit)

• svcconf – Optional conf for the Storm service

• outp – A output helper for the Cmdr instance

Notes

Both the CmdrCore and Storm service proxy should be fini()’d for proper teardown

Returns

A CmdrCore instance and proxy to the Storm service

Return type

(CmdrCore, Proxy)

async synapse.lib.jupyter.getTempCoreProx(mods=None)

Get a Telepath Proxt to a Cortex instance which is backed by a temporary Cortex.

Parameters

mods (list) – A list of additional CoreModules to load in the Cortex.

Notes

The Proxy returned by this should be fini()’d to tear down the temporary Cortex.

Returns

s_telepath.Proxy

synapse.lib.layer module

The Layer 2.0 archtecture introduces several optimized node/message serialization formats used by the layers to optimize returning primitives and facilitate efficient node construction:

Note

This interface is subject to change between minor revisions.

Storage Types (<stortype>)

In Layers 2.0, each node property from the model has an associated “storage type”. Each storage type determines how the data is indexed and represented within the Layer. This formalizes the separation of “data model” from “storage model”. Each data model type has a “stortype” property which coresponds to one of the STOR_TYPE_XXX values. The knowledge of the mapping of data model types to storage types is the responsibility of the data model, making the Layer implementation fully decoupled from the data model.

Node Edits / Edits

A node edit consists of a (<buid>, <form>, [edits]) tuple. An edit is Tuple of (<type>, <info>, List[NodeEdits]) where the first element is an int that matches to an EDIT_* constant below, the info is a tuple that varies depending on the first element, and the third element is a list of dependent NodeEdits that will only be applied if the edit actually makes a change.

Storage Node (<sode>)

A storage node is a layer/storage optimized node representation which is similar to a “packed node”. A storage node may be partial ( as it is produced by a given layer ) and are joined by the view/snap into “full” storage nodes which are used to construct Node() instances.

Sode format:

(<buid>, {

    'ndef': (<formname>, <formvalu>),

    'props': {
        <propname>: <propvalu>,
    }

    'tags': {
        <tagname>: <tagvalu>,
    }

    'tagprops: {
        <tagname>: {
            <propname>: <propvalu>,
        },
    }

    # changes that were *just* made.
    'edits': [
        <edit>
    ]

}),

class synapse.lib.layer.IndxBy(layr, abrv, db)

Bases: object

IndxBy sub-classes encapsulate access methods and encoding details for various types of properties within the layer to be lifted/compared by storage types.

buidsByDups(indx)

buidsByPref(indx=b'')

buidsByRange(minindx, maxindx)

buidsByRangeBack(minindx, maxindx)

getNodeValu(buid)

hasIndxBuid(indx, buid)

keyBuidsByDups(indx)

keyBuidsByPref(indx=b'')

keyBuidsByRange(minindx, maxindx)

keyBuidsByRangeBack(minindx, maxindx)

Yields backwards from maxindx to minindx

scanByDups(indx)

scanByPref(indx=b'')

scanByRange(minindx, maxindx)

class synapse.lib.layer.IndxByForm(layr, form)

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)

class synapse.lib.layer.IndxByProp(layr, form, prop)

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)

class synapse.lib.layer.IndxByPropArray(layr, form, prop)

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)

class synapse.lib.layer.IndxByTag(layr, form, tag)

Bases: synapse.lib.layer.IndxBy

getNodeValuForm(buid)

class synapse.lib.layer.IndxByTagProp(layr, form, tag, prop)

Bases: synapse.lib.layer.IndxBy

getNodeValu(buid)

class synapse.lib.layer.Layer

Bases: synapse.lib.nexus.Pusher

The base class for a cortex layer.

async clone(newdirn)

Copy the contents of this layer to a new layer

async delete()

Delete the underlying storage

getAbrvProp(abrv)

async getEdgeVerbs()

async getEdges(verb=None)

async getEditIndx()

Returns what will be the next (i.e. 1 past the last) nodeedit log index.

async getEditOffs()

Return the offset of the last recorded log entry. Returns -1 if nodeedit log is disabled or empty.

async getEditSize()

async getFormCounts()

getFormProps()

getIdenFutu(iden=None)

async getLayerSize()

Get the total storage size for the layer.

async getMirrorStatus()

async getModelVers()

async getNodeData(buid, name)

Return a single element of a buid’s node data

getNodeEditWindow()

async getNodeForm(buid)

async getNodeTag(buid, tag)

async getNodeValu(buid, prop=None)

Retrieve either the form valu or a prop valu for the given node by buid.

getPropAbrv(form, prop)

async getPropCount(formname, propname=None, maxsize=None)

Return the number of property rows in the layer for the given form/prop.

getStorIndx(stortype, valu)

async getStorNode(buid)

async getStorNodes()

Yield (buid, sode) tuples for all the nodes with props/tags/tagprops stored in this layer.

async getTagCount(tagname, formname=None)

Return the number of tag rows in the layer for the given tag/form.

getTagPropAbrv(*args)

getTagProps()

async getUnivPropCount(propname, maxsize=None)

Return the number of universal property rows in the layer for the given prop.

async hasNodeData(buid, name)

async hasNodeEdge(buid1, verb, buid2)

async hasTagProp(name)

async initLayerActive()

async initLayerPassive()

async initUpstreamSync(url)

async iterFormRows(form, stortype=None, startvalu=None)

Yields buid, valu tuples of nodes of a single form, optionally (re)starting at startvalu.

Parameters

• form (str) – A form name.

• stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop

• startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns

AsyncIterator[Tuple(buid, valu)]

async iterLayerNodeEdits()

Scan the full layer and yield artificial sets of nodeedits.

async iterNodeData(buid)

Return a generator of all a buid’s node data

async iterNodeEdgesN1(buid, verb=None)

async iterNodeEdgesN2(buid, verb=None)

async iterNodeEditLog(offs=0)

Iterate the node edit log and yield (offs, edits, meta) tuples.

async iterNodeEditLogBack(offs=0)

Iterate the node edit log and yield (offs, edits, meta) tuples in reverse.

async iterPropRows(form, prop, stortype=None, startvalu=None)

Yields buid, valu tuples of nodes with a particular secondary property, optionally (re)starting at startvalu.

Parameters

• form (str) – A form name.

• prop (str) – A universal property name.

• stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop

• startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns

AsyncIterator[Tuple(buid, valu)]

async iterTagPropRows(tag, prop, form=None, stortype=None, startvalu=None)

Yields (buid, valu) that match a tag:prop, optionally (re)starting at startvalu.

Parameters

• tag (str) – tag name

• prop (str) – prop name

• form (Optional[str]) – optional form name

• stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop

• startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns

AsyncIterator[Tuple(buid, valu)]

async iterTagRows(tag, form=None, starttupl=None)

Yields (buid, (valu, form)) values that match a tag and optional form, optionally (re)starting at starttupl.

Parameters

• tag (str) – the tag to match

• form (Optional[str]) – if present, only yields buids of nodes that match the form.

• starttupl (Optional[Tuple[buid, form]]) – if present, (re)starts the stream of values there.

Returns

AsyncIterator[Tuple(buid, (valu, form))]

Note

This yields (buid, (tagvalu, form)) instead of just buid, valu in order to allow resuming an interrupted call by feeding the last value retrieved into starttupl

async iterUnivRows(prop, stortype=None, startvalu=None)

Yields buid, valu tuples of nodes with a particular universal property, optionally (re)starting at startvalu.

Parameters

• prop (str) – A universal property name.

• stortype (Optional[int]) – a STOR_TYPE_* integer representing the type of form:prop

• startvalu (Any) – The value to start at. May only be not None if stortype is not None.

Returns

AsyncIterator[Tuple(buid, valu)]

async liftByDataName(name)

async liftByFormValu(form, cmprvals)

async liftByProp(form, prop)

async liftByPropArray(form, prop, cmprvals)

async liftByPropValu(form, prop, cmprvals)

async liftByTag(tag, form=None)

async liftByTagProp(form, tag, prop)

async liftByTagPropValu(form, tag, prop, cmprvals)

Note: form may be None

async liftByTagValu(tag, cmpr, valu, form=None)

async liftTagProp(name)

async makeSplices(offs, nodeedits, meta, reverse=False)

Flatten a set of nodeedits into splices.

mayDelBuid(buid, sode)

nodeeditctor

alias of synapse.lib.slabseqn.SlabSeqn

async pack()

async saveNodeEdits(edits, meta)

Save node edits to the layer and return a tuple of (nexsoffs, changes).

Note: nexsoffs will be None if there are no changes.

async setLayerInfo(name, valu)

Set a mutable layer property.

async setModelVers(vers)

setPropAbrv(form, prop)

setSodeDirty(buid, sode, form)

setTagPropAbrv(*args)

async splices(offs=None, size=None)

Yield (offs, splice) tuples from the nodeedit log starting from the given offset.

Nodeedits will be flattened into splices before being yielded.

async splicesBack(offs=None, size=None)

async stat()

async storNodeEdits(nodeedits, meta)

async storNodeEditsNoLift(nodeedits, meta)

Execute a series of node edit operations.

Does not return the updated nodes.

async syncIndexEvents(offs, matchdef, wait=True)

Yield (offs, (buid, form, ETYPE, VALS, META)) tuples from the nodeedit log starting from the given offset. Only edits that match the filter in matchdef will be yielded.

Notes

ETYPE is an constant EDIT_* above. VALS is a tuple whose format depends on ETYPE, outlined in the comment next to the constant. META is a dict that may contain keys ‘user’ and ‘time’ to represent the iden of the user that initiated the change, and the time that it took place, respectively.

Additionally, every 1000 entries, an entry (offs, (None, None, EDIT_PROGRESS, (), ())) message is emitted.

The matchdef dict may contain the following keys: forms, props, tags, tagprops. The value must be a sequence of strings. Each key/val combination is treated as an “or”, so each key and value yields more events. forms: EDIT_NODE_ADD and EDIT_NODE_DEL events. Matches events for nodes with forms in the value list. props: EDIT_PROP_SET and EDIT_PROP_DEL events. Values must be in form:prop or .universal form tags: EDIT_TAG_SET and EDIT_TAG_DEL events. Values must be the raw tag with no #. tagprops: EDIT_TAGPROP_SET and EDIT_TAGPROP_DEL events. Values must be just the prop or tag:prop.

Will not yield any values if this layer was not created with logedits enabled

Parameters

• offs (int) – starting nexus/editlog offset

• matchdef (Dict[str, Sequence[str]]) – a dict describing which events are yielded

• wait (bool) – whether to pend and stream value until this layer is fini’d

async syncNodeEdits(offs, wait=True)

Identical to syncNodeEdits2, but doesn’t yield meta

async syncNodeEdits2(offs, wait=True)

Once caught up with storage, yield them in realtime.

Returns

Tuple of offset(int), nodeedits, meta(dict)

async truncate()

Nuke all the contents in the layer, leaving an empty layer

async verify(config=None)

async verifyAllBuids(scanconf=None)

async verifyAllProps(scanconf=None)

async verifyAllTagProps(scanconf=None)

async verifyAllTags(scanconf=None)

async verifyBuidTag(buid, formname, tagname, tagvalu)

async verifyByBuid(buid, sode)

async verifyByProp(form, prop, autofix=None)

async verifyByPropArray(form, prop, autofix=None)

async verifyByTag(tag, autofix=None)

async verifyByTagProp(form, tag, prop, autofix=None)

async waitEditOffs(offs, timeout=None)

Wait for the node edit log to write an entry at/past the given offset.

async waitForHot()

Wait for the layer’s slab to be prefaulted and locked into memory if lockmemory is true, otherwise return.

async waitUpstreamOffs(iden, offs)

class synapse.lib.layer.LayerApi

Bases: synapse.lib.cell.CellApi

async getEditIndx()

Returns what will be the next nodeedit log index.

async getEditSize()

Return the total number of (edits, meta) pairs in the layer changelog.

async getIden()

async iterLayerNodeEdits()

Scan the full layer and yield artificial nodeedit sets.

saveNodeEdits(edits, meta)

Save node edits to the layer and return a tuple of (nexsoffs, changes).

Note: nexsoffs will be None if there are no changes.

async splices(offs=None, size=None)

Yield (offs, splice) tuples from the nodeedit log starting from the given offset.

Nodeedits will be flattened into splices before being yielded.

async storNodeEdits(nodeedits, meta=None)

async storNodeEditsNoLift(nodeedits, meta=None)

async syncNodeEdits(offs, wait=True)

Yield (offs, nodeedits) tuples from the nodeedit log starting from the given offset.

Once caught up with storage, yield them in realtime.

async syncNodeEdits2(offs, wait=True)

class synapse.lib.layer.StorType(layr, stortype)

Bases: object

decodeIndx(valu)

indx(valu)

async indxBy(liftby, cmpr, valu)

async indxByForm(form, cmpr, valu)

async indxByProp(form, prop, cmpr, valu)

async indxByPropArray(form, prop, cmpr, valu)

async indxByTagProp(form, tag, prop, cmpr, valu)

async verifyBuidProp(buid, form, prop, valu)

class synapse.lib.layer.StorTypeFloat(layr, stortype, size=8)

Bases: synapse.lib.layer.StorType

FloatPackNegMax = b'\x80\x00\x00\x00\x00\x00\x00\x00'

FloatPackNegMin = b'\xff\xf0\x00\x00\x00\x00\x00\x00'

FloatPackPosMax = b'\x7f\xf0\x00\x00\x00\x00\x00\x00'

FloatPackPosMin = b'\x00\x00\x00\x00\x00\x00\x00\x00'

FloatPacker = <Struct object>

decodeIndx(bytz)

fpack()

S.pack(v1, v2, …) -> bytes

Return a bytes object containing values v1, v2, … packed according to the format string S.format. See help(struct) for more on format strings.

indx(valu)

class synapse.lib.layer.StorTypeFqdn(layr)

Bases: synapse.lib.layer.StorTypeUtf8

decodeIndx(bytz)

indx(norm)

class synapse.lib.layer.StorTypeGuid(layr)

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

indx(valu)

class synapse.lib.layer.StorTypeHier(layr, stortype, sepr='.')

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

getHierIndx(valu)

indx(valu)

class synapse.lib.layer.StorTypeHugeNum(layr, stortype)

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

getHugeIndx(norm)

indx(norm)

class synapse.lib.layer.StorTypeInt(layr, stortype, size, signed)

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

getIntIndx(valu)

indx(valu)

class synapse.lib.layer.StorTypeIpv6(layr)

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

getIPv6Indx(valu)

indx(valu)

class synapse.lib.layer.StorTypeIval(layr)

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

indx(valu)

class synapse.lib.layer.StorTypeLatLon(layr)

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

indx(valu)

class synapse.lib.layer.StorTypeLoc(layr)

Bases: synapse.lib.layer.StorTypeHier

class synapse.lib.layer.StorTypeMsgp(layr)

Bases: synapse.lib.layer.StorType

indx(valu)

class synapse.lib.layer.StorTypeTag(layr)

Bases: synapse.lib.layer.StorTypeHier

static getTagFilt(cmpr, valu)

class synapse.lib.layer.StorTypeTime(layr)

Bases: synapse.lib.layer.StorTypeInt

class synapse.lib.layer.StorTypeUtf8(layr)

Bases: synapse.lib.layer.StorType

decodeIndx(bytz)

indx(valu)

synapse.lib.layer.getFlatEdits(nodeedits)

synapse.lib.link module

class synapse.lib.link.Link

Bases: synapse.lib.base.Base

A Link() is created to wrap a socket reader/writer.

feed(byts)

Used by Plex() to unpack bytes.

get(name, defval=None)

Get a property from the Link info.

getAddrInfo()

Get a summary of address information related to the link.

async getSpawnInfo()

getTlsPeerCn()

async recv(size)

async recvsize(size)

async rx()

async send(byts)

set(name, valu)

Set a property in the Link info.

async tx(mesg)

Async transmit routine which will wait for writer drain().

async synapse.lib.link.connect(host, port, ssl=None, hostname=None)

Async connect and return a Link().

async synapse.lib.link.fromspawn(spawninfo)

async synapse.lib.link.linkfile(mode='wb')

Connect a socketpair to a file-object and return (link, file).

async synapse.lib.link.linksock()

Connect a Link, socket pair.

async synapse.lib.link.listen(host, port, onlink, ssl=None)

Listen on the given host/port and fire onlink(Link).

Returns a server object that contains the listening sockets

async synapse.lib.link.unixconnect(path)

Connect to a PF_UNIX server listening on the given path.

async synapse.lib.link.unixlisten(path, onlink)

Start an PF_UNIX server listening on the given path.

synapse.lib.lmdbslab module

class synapse.lib.lmdbslab.GuidStor(slab, name)

Bases: object

gen(iden)

class synapse.lib.lmdbslab.Hist(slab, name)

Bases: object

A class for storing items in a slab by time.

Each added item is inserted into the specified db within the slab using the current epoch-millis time stamp as the key.

add(item, tick=None)

carve(tick, tock=None)

class synapse.lib.lmdbslab.HotCount

Bases: synapse.lib.lmdbslab.HotKeyVal

Like HotKeyVal, but optimized for integer/count vals

static DecFunc(b)

Decode a signed 64-bit int from 8 byte big-endian

static EncFunc(i)

Encode a signed 64-bit int into 8 byte big-endian bytes

get(name: str, defv=0)

inc(name: str, valu=1)

set(name: str, valu)

class synapse.lib.lmdbslab.HotKeyVal

Bases: synapse.lib.base.Base

A hot-loop capable keyval that only syncs on commit.

static DecFunc(byts)

Use msgpack to de-serialize a python object.

Parameters

byts (bytes) – The bytes to de-serialize

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Returns

The de-serialized object

Return type

obj

static EncFunc(item)

Use msgpack to serialize a compatible python object.

Parameters

item (obj) – The object to serialize

Notes

String objects are encoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow encoding bad input strings.

Returns

The serialized bytes in msgpack format.

Return type

bytes

delete(name: str)

get(name: str, defv=None)

pack()

set(name: str, valu)

sync()

class synapse.lib.lmdbslab.MultiQueue

Bases: synapse.lib.base.Base

Allows creation/consumption of multiple durable queues in a slab.

async add(name, info)

async cull(name, offs)

Remove up-to (and including) the queue entry at offs.

async dele(name, minoffs, maxoffs)

Remove queue entries from minoffs, up-to (and including) the queue entry at maxoffs.

exists(name)

async get(name, offs, wait=False, cull=True)

Return (nextoffs, item) tuple or (-1, None) for the given offset.

async gets(name, offs, size=None, cull=False, wait=False)

Yield (offs, item) tuples from the message queue.

list()

offset(name)

async pop(name, offs)

Pop a single entry from the named queue by offset.

async put(name, item, reqid=None)

async puts(name, items, reqid=None)

async rem(name)

async sets(name, offs, items)

Overwrite queue entries with the values in items, starting at offs.

size(name)

status(name)

class synapse.lib.lmdbslab.Scan(slab, db)

Bases: object

A state-object used by Slab. Not to be instantiated directly.

Parameters

• slab (Slab) – which slab the scan is over

• db (str) – name of open database on the slab

bump()

first()

isatitem()

Returns if the cursor is at the value in atitem

iterfunc()

iternext()

resume()

set_key(lkey)

set_range(lkey, valu=None)

class synapse.lib.lmdbslab.ScanBack(slab, db)

Bases: synapse.lib.lmdbslab.Scan

A state-object used by Slab. Not to be instantiated directly.

Scans backwards.

first()

iterfunc()

resume()

set_key(lkey)

set_range(lkey)

class synapse.lib.lmdbslab.ScanKeys(slab, db)

Bases: synapse.lib.lmdbslab.Scan

An iterator over the keys of the database. If the database is dupsort, a key with multiple values with be yielded once for each value.

isatitem()

Returns if the cursor is at the value in atitem

iterfunc()

iternext()

resume()

class synapse.lib.lmdbslab.Slab

Bases: synapse.lib.base.Base

A “monolithic” LMDB instance for use in a asyncio loop thread.

COMMIT_PERIOD = 0.2

DEFAULT_GROWSIZE = None

DEFAULT_MAPSIZE = 1073741824

WARN_COMMIT_TIME_MS = 1000

allslabs = {}

copydb(sourcedbname, destslab, destdbname=None, progresscb=None)

Copy an entire database in this slab to a new database in potentially another slab.

Parameters

• sourcedbname (str) – name of the db in the source environment

• destslab (LmdbSlab) – which slab to copy rows to

• destdbname (str) – the name of the database to copy rows to in destslab

• progresscb (Callable[int]) – if not None, this function will be periodically called with the number of rows completed

Returns

the number of rows copied

Return type

(int)

Note

If any rows already exist in the target database, this method returns an error. This means that one cannot use destdbname=None unless there are no explicit databases in the destination slab.

async copyslab(dstpath, compact=True)

async countByPref(byts, db=None, maxsize=None)

Return the number of rows in the given db with the matching prefix bytes.

dbexists(name)

The DB exists already if there’s a key in the default DB with the name of the database

delete(lkey, val=None, db=None)

dropdb(name)

Deletes an entire database (i.e. a table), losing all data.

async fini()

Shut down the object and notify any onfini() coroutines.

Returns

Remaining ref count

firstkey(db=None)

Return the first key or None from the given db.

forcecommit()

Note

This method may raise a MapFullError

get(lkey, db=None)

async getHotCount(name)

async getMultiQueue(name, nexsroot=None)

getNameAbrv(name)

getSeqn(name)

async classmethod getSlabStats()

classmethod getSlabsInDir(dirn)

Returns all open slabs under a directory

has(lkey, db=None)

hasdup(lkey, lval, db=None)

async classmethod initSyncLoop(inst)

initdb(name, dupsort=False, integerkey=False, dupfixed=False)

last(db=None)

Return the last key/value pair from the given db.

lastkey(db=None)

Return the last key or None from the given db.

pop(lkey, db=None)

prefexists(byts, db=None)

Returns True if a prefix exists in the db.

put(lkey, lval, dupdata=False, overwrite=True, append=False, db=None)

putmulti(kvpairs, dupdata=False, append=False, db=None)

Returns

Tuple of number of items consumed, number of items added

rangeexists(lmin, lmax=None, db=None)

Returns True if at least one key exists in the range.

replace(lkey, lval, db=None)

Like put, but returns the previous value if existed

scanByDups(lkey, db=None)

scanByDupsBack(lkey, db=None)

scanByFull(db=None)

scanByFullBack(db=None)

scanByPref(byts, startkey=None, startvalu=None, db=None)

Parameters

• byts (bytes) – prefix to match on

• startkey (Optional[bytes]) – if present, will start scanning at key=byts+startkey

• startvalu (Optional[bytes]) – if present, will start scanning at (key+startkey, startvalu)

Notes

startvalu only makes sense if byts+startkey matches an entire key. startvalu is only value for dupsort=True dbs

scanByPrefBack(byts, db=None)

scanByRange(lmin, lmax=None, db=None)

scanByRangeBack(lmax, lmin=None, db=None)

scanKeys(db=None)

scanKeysByPref(byts, db=None)

stat(db=None)

statinfo()

async sync()

async classmethod syncLoopOnce()

async classmethod syncLoopTask()

syncevnt = None

synctask = None

async trash()

Deletes underlying storage

class synapse.lib.lmdbslab.SlabAbrv(slab, name)

Bases: object

A utility for translating arbitrary bytes into fixed with id bytes

abrvToByts(abrv)

abrvToName(byts)

bytsToAbrv(byts)

keys()

nameToAbrv(name)

names()

setBytsToAbrv(byts)

class synapse.lib.lmdbslab.SlabDict(slab, db=None, pref=b'')

Bases: object

A dictionary-like object which stores its props in a slab via a prefix.

It is assumed that only one SlabDict with a given prefix exists at any given time, but it is up to the caller to cache them.

get(name, defval=None)

Get a name from the SlabDict.

Parameters

• name (str) – The key name.

• defval (obj) – The default value to return.

Returns

The return value, or None.

Return type

(obj)

inc(name, valu=1)

items()

Return a tuple of (prop, valu) tuples from the SlabDict.

Returns

Tuple of (name, valu) tuples.

Return type

(((str, object), …))

keys()

pop(name, defval=None)

Pop a name from the SlabDict.

Parameters

• name (str) – The name to remove.

• defval (obj) – The default value to return if the name is not present.

Returns

The object stored in the SlabDict, or defval if the object was not present.

Return type

object

set(name, valu)

Set a name in the SlabDict.

Parameters

• name (str) – The key name.

• valu (obj) – A msgpack compatible value.

Returns

None

synapse.lib.modelrev module

class synapse.lib.modelrev.ModelRev(core)

Bases: object

async revCoreLayers()

async revModel20210126(layers)

async revModel20210312(layers)

async revModel20210528(layers)

async revModel20210801(layers)

async revModel20211112(layers)

async revModel20220307(layers)

async revModel20220315(layers)

async runStorm(text, opts=None)

Run storm code in a schedcoro and log the output messages.

Parameters

• text (str) – Storm query to execute.

• opts – Storm opts.

Returns

None

synapse.lib.module module

class synapse.lib.module.CoreModule(core, conf=None)

Bases: object

confdefs = ()

getConfPath()

Get the path to the module specific config file (conf.yaml).

Notes

This creates the parent directory for the conf.yaml file if it does not exist. This API exists to allow a implementor to get the conf path during initCoreModule and drop a example config if needed. One use case of that is for missing configuration values, an example config can be written to the file and a exception raised.

Returns

Path to where the conf file is located at.

Return type

str

getModDir()

Get the path to the module specific directory.

Notes

This creates the directory if it did not previously exist.

Returns

The filepath to the module specific directory.

Return type

str

getModName()

Return the lowercased name of this module.

Notes

This pulls the mod_name attribute on the class. This allows an implementer to set a arbitrary name for the module. If this attribute is not set, it defaults to self.__class__.__name__.lower() and sets mod_name to that value.

Returns

The module name.

Return type

(str)

getModPath(*paths)

Construct a path relative to this module’s working directory.

Parameters

*paths – A list of path strings

Notes

This creates the module specific directory if it does not exist.

Returns

The full path (or None if no cortex dir is configured).

Return type

(str)

getModelDefs()

getStormCmds()

Module implementers may override this to provide a list of Storm commands which will be loaded into the Cortex.

Returns

A list of Storm Command classes (not instances).

Return type

list

async initCoreModule()

Module implementers may override this method to initialize the module after the Cortex has completed and is accessible to perform storage operations.

Notes

This is the preferred function to override for implementing custom code that needs to be executed during Cortex startup.

Any exception raised within this method will remove the module from the list of currently loaded modules.

This is called for modules after getModelDefs() and getStormCmds() has been called, in order to allow for model loading and storm command loading prior to code execution offered by initCoreModule.

A failure during initCoreModule will not unload data model or storm commands registered by the module.

Returns

None

mod_name = None

async preCoreModule()

Module implementers may override this method to execute code immediately after a module has been loaded.

Notes

The initCoreModule function is preferred for overriding instead of preCoreModule().

No Cortex layer/storage operations will function in preCoreModule.

Any exception raised within this method will halt additional loading of the module.

Returns

None

synapse.lib.modules module

Module which implements the synapse module API/convention.

synapse.lib.msgpack module

class synapse.lib.msgpack.Unpk

Bases: object

An extension of the msgpack streaming Unpacker which reports sizes.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

feed(byts)

Feed bytes to the unpacker and return completed objects.

Parameters

byts (bytes) – Bytes to unpack.

Notes

It is intended that this function is called multiple times with bytes from some sort of a stream, as it will unpack and return objects as they are available.

Returns

List of tuples containing the item size and the unpacked item.

Return type

list

synapse.lib.msgpack.deepcopy(item)

Copy a msgpack serializable by packing then unpacking it. For complex primitives, this runs in about 1/3 the time of copy.deepcopy()

synapse.lib.msgpack.dumpfile(item, path)

Dump an object to a file by path.

Parameters

• item (object) – The object to serialize.

• path (str) – The file path to save.

Returns

None

synapse.lib.msgpack.en(item)

Use msgpack to serialize a compatible python object.

Parameters

item (obj) – The object to serialize

Notes

String objects are encoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow encoding bad input strings.

Returns

The serialized bytes in msgpack format.

Return type

bytes

synapse.lib.msgpack.isok(item)

Returns True if the item can be msgpacked (by testing packing).

synapse.lib.msgpack.iterfd(fd)

Generator which unpacks a file object of msgpacked content.

Parameters

fd – File object to consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Yields

Objects from a msgpack stream.

synapse.lib.msgpack.iterfile(path, since=- 1)

Generator which yields msgpack objects from a file path.

Parameters

path – File path to open and consume data from.

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Yields

Objects from a msgpack stream.

synapse.lib.msgpack.loadfile(path)

Load and upack the msgpack bytes from a file by path.

Parameters

path (str) – The file path to a message pack file.

Raises

msgpack.exceptions.ExtraData – If the file contains multiple objects.

Returns

The decoded python object.

Return type

(obj)

synapse.lib.msgpack.un(byts)

Use msgpack to de-serialize a python object.

Parameters

byts (bytes) – The bytes to de-serialize

Notes

String objects are decoded using utf8 encoding. In order to handle potentially malformed input, unicode_errors='surrogatepass' is set to allow decoding bad input strings.

Returns

The de-serialized object

Return type

obj

synapse.lib.multislabseqn module

class synapse.lib.multislabseqn.MultiSlabSeqn

Bases: synapse.lib.base.Base

An append-optimized sequence of byte blobs stored across multiple slabs for fast rotating/culling

async add(item: Any, indx=None) → int

Add a single item to the sequence.

async cull(offs: int) → bool

Remove entries up to (and including) the given offset.

async get(offs: int) → Any

Retrieve a single row by offset

getOffsetEvent(offs: int) → asyncio.locks.Event

Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.

async gets(offs, wait=True) → AsyncIterator[Tuple[int, Any]]

Just like iter, but optionally waits for new entries once the end is reached.

index() → int

Return the current index to be used

async iter(offs: int) → AsyncIterator[Tuple[int, Any]]

Iterate over items in a sequence from a given offset.

Parameters

offs (int) – The offset to begin iterating from.

Yields

(indx, valu) – The index and valu of the item.

async last() → Optional[Tuple[int, Any]]

async rotate() → int

Rotate the Nexus log at the current index.

Note

After this executes the tailseqn will be empty. Waiting for this indx to be written will indicate when it is possible to cull 1 minus the return value such that the rotated seqn is deleted.

Returns

The starting index of the new seqn

Return type

int

setIndex(indx: int) → None

static slabFilename(dirn: str, indx: int)

async waitForOffset(offs: int, timeout=None) → bool

Returns

true if the event got set, False if timed out

synapse.lib.nexus module

class synapse.lib.nexus.ChangeDist

Bases: synapse.lib.base.Base

A utility class to distribute new change entries to mirrors/followers

update() → bool

class synapse.lib.nexus.NexsRoot

Bases: synapse.lib.base.Base

async cull(offs)

async eat(nexsiden, event, args, kwargs, meta)

Actually mutate for the given nexsiden instance.

getChangeDist(offs: int) → AsyncIterator[synapse.lib.nexus.ChangeDist]

async index()

async issue(nexsiden, event, args, kwargs, meta=None)

If I’m not a follower, mutate, otherwise, ask the leader to make the change and wait for the follower loop to hand me the result through a future.

async iter(offs: int) → AsyncIterator[Any]

Returns an iterator of change entries in the log

async promote()

async recover() → None

Replays the last entry in the nexus log in case we crashed between writing the log and applying it.

Notes

This must be called at cell startup after subsystems are initialized but before any write transactions might happen.

The log can only have recorded 1 entry ahead of what is applied. All log actions are idempotent, so replaying the last action that (might have) already happened is harmless.

async rotate()

async runMirrorLoop(proxy)

async setindex(indx)

async startup(mirurl, celliden=None)

async waitOffs(offs, timeout=None)

class synapse.lib.nexus.Pusher

Bases: synapse.lib.base.Base

A mixin-class to manage distributing changes where one might plug in mirroring or consensus protocols

classmethod onPush(event: str, passitem=False) → Callable

Decorator that registers a method to be a handler for a named event

Parameters

• event – string that distinguishes one handler from another. Must be unique per Pusher subclass

• passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”

classmethod onPushAuto(event: str, passitem=False) → Callable

Decorator that does the same as onPush, except automatically creates the top half method

Parameters

• event – string that distinguishes one handler from another. Must be unique per Pusher subclass

• passitem – whether to pass the (offs, mesg) tuple to the handler as “nexsitem”

async saveToNexs(name, *args, **kwargs)

setNexsRoot(nexsroot)

class synapse.lib.nexus.RegMethType(name: str, bases: List[type], attrs: Dict[str, Any])

Bases: type

Metaclass that collects all methods in class with _regme prop into a class member called _regclstupls

synapse.lib.node module

class synapse.lib.node.Node(snap, sode, bylayer=None)

Bases: object

A Cortex hypergraph node.

NOTE: This object is for local Cortex use during a single Xact.

async addEdge(verb, n2iden)

async addTag(tag, valu=(None, None))

Add a tag to a node.

Parameters

• tag (str) – The tag to add to the node.

• valu – The optional tag value. If specified, this must be a value that norms as a valid time interval as an ival.

Returns

This returns None.

Return type

None

async delEdge(verb, n2iden)

async delTag(tag, init=False)

Delete a tag from the node.

async delTagProp(tag, name)

async delete(force=False)

Delete a node from the cortex.

The following tear-down operations occur in order:

• validate that you have permissions to delete the node

• validate that you have permissions to delete all tags

• validate that there are no remaining references to the node.

• delete all the tags (bottom up)

  • fire onDelTag() handlers

  • delete tag properties from storage

  • log tag:del splices

• delete all secondary properties

  • fire onDelProp handler

  • delete secondary property from storage

  • log prop:del splices

• delete the primary property

  • fire onDel handlers for the node

  • delete primary property from storage

  • log node:del splices

async filter(runt, text, opts=None, path=None)

get(name)

Return a secondary property value from the Node.

Parameters

name (str) – The name of a secondary property.

Returns

The secondary property value or None.

Return type

(obj)

getByLayer()

Return a dictionary that translates the node’s bylayer dict to a primitive.

async getData(name, defv=None)

async getEmbeds(embeds)

Return a dictionary of property embeddings.

getNodeRefs()

Return a list of (prop, (form, valu)) refs out for the node.

async getStorNodes()

Return a list of the raw storage nodes for each layer.

getTag(name, defval=None)

getTagProp(tag, prop, defval=None)

Return the value (or defval) of the given tag property.

getTagProps(tag)

getTags(leaf=False)

has(name)

async hasData(name)

hasTag(name)

hasTagProp(tag, prop)

Check if a #foo.bar:baz tag property exists on the node.

iden()

async iterData()

async iterEdgesN1(verb=None)

async iterEdgesN2(verb=None)

pack(dorepr=False)

Return the serializable/packed version of the node.

Parameters

dorepr (bool) – Include repr information for human readable versions of properties.

Returns

An (ndef, info) node tuple.

Return type

(tuple)

async pop(name, init=False)

Remove a property from a node and return the value

async popData(name)

repr(name=None, defv=None)

reprs()

Return a dictionary of repr values for props whose repr is different than the system mode value.

async seen(tick, source=None)

Update the .seen interval and optionally a source specific seen node.

async set(name, valu, init=False)

Set a property on the node.

Parameters

• name (str) – The name of the property.

• valu (obj) – The value of the property.

• init (bool) – Set to True to disable read-only enforcement

Returns

True if the property was changed.

Return type

(bool)

async setData(name, valu)

async setTagProp(tag, name, valu)

Set the value of the given tag property.

async storm(runt, text, opts=None, path=None)

Parameters

path (Path) – If set, then vars from path are copied into the new runtime, and vars are copied back out into path at the end

Note

If opts is not None and opts[‘vars’] is set and path is not None, then values of path vars take precedent

tagpropreprs()

Return a dictionary of repr values for tagprops whose repr is different than the system mode value.

class synapse.lib.node.Path(vars, nodes)

Bases: object

A path context tracked through the storm runtime.

clone()

finiframe()

Pop a scope frame from the path, restoring runt if at the top :param runt: A storm runtime to restore if we’re at the top :type runt: Runtime :param merge: Set to true to merge vars back up into the next frame :type merge: bool

fork(node)

getVar(name, defv=<synapse.common.NoValu object>)

initframe(initvars=None)

meta(name, valu)

Add node specific metadata to be returned with the node.

async pack(path=False)

async popVar(name)

async setVar(name, valu)

synapse.lib.node.iden(pode)

Return the iden (buid) of the packed node.

Parameters

pode (tuple) – A packed node.

Returns

The node iden.

Return type

str

synapse.lib.node.ndef(pode)

Return a node definition (<form>,<valu>) tuple from the node.

Parameters

pode (tuple) – A packed node.

Returns

The (<form>,<valu>) tuple for the node

Return type

((str,obj))

synapse.lib.node.prop(pode, prop)

Return the valu of a given property on the node.

Parameters

• pode (tuple) – A packed node.

• prop (str) – Property to retrieve.

Notes

The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).

Returns:

synapse.lib.node.props(pode)

Get the props from the node.

Parameters

pode (tuple) – A packed node.

Notes

This will include any universal props present on the node.

Returns

A dictionary of properties.

Return type

dict

synapse.lib.node.reprNdef(pode)

Get the ndef of the pode with a human readable value.

Parameters

pode (tuple) – A packed node.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

Returns

A tuple of form and the human readable value.

Return type

(str, str)

synapse.lib.node.reprProp(pode, prop)

Get the human readable value for a secondary property from the pode.

Parameters

• pode (tuple) – A packed node.

• prop –

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

The prop argument may be the full property name (foo:bar:baz), relative property name (:baz) , or the unadorned property name (baz).

Returns

The human readable property value. If the property is not present, returns None.

Return type

str

synapse.lib.node.reprTag(pode, tag)

Get the human readable value for the tag timestamp from the pode.

Parameters

• pode (tuple) – A packed node.

• tag (str) – The tag to get the value for.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

If the tag does not have a timestamp, this returns a empty string.

Returns

The human readable value for the tag. If the tag is not present, returns None.

Return type

str

synapse.lib.node.reprTagProps(pode, tag)

Get the human readable values for any tagprops on a tag for a given node.

Parameters

• pode (tuple) – A packed node.

• tag (str) – The tag to get the tagprops reprs for.

Notes

The human readable value is only available if the node came from a storm query execution where the repr key was passed into the opts argument with a True value.

If the tag does not have any tagprops associated with it, this returns an empty list.

Returns

A list of tuples, containing the name of the tagprop and the repr value.

Return type

list

synapse.lib.node.tagged(pode, tag)

Check if a packed node has a given tag.

Parameters

• pode (tuple) – A packed node.

• tag (str) – The tag to check.

Examples

Check if a node is tagged with “woot” and dostuff if it is.

if s_node.tagged(node,’woot’):

dostuff()

Notes

If the tag starts with #, this is removed prior to checking.

Returns

True if the tag is present. False otherwise.

Return type

bool

synapse.lib.node.tags(pode, leaf=False)

Get all the tags for a given node.

Parameters

• pode (tuple) – A packed node.

• leaf (bool) – If True, only return leaf tags

Returns

A list of tag strings.

Return type

list

synapse.lib.node.tagsnice(pode)

Get all the leaf tags and the tags that have values or tagprops.

Parameters

pode (tuple) – A packed node.

Returns

A list of tag strings.

Return type

list

synapse.lib.output module

Tools for easily hookable output from cli-like tools.

class synapse.lib.output.OutPut

Bases: object

printf(mesg, addnl=True)

class synapse.lib.output.OutPutBytes

Bases: synapse.lib.output.OutPutFd

class synapse.lib.output.OutPutFd(fd, enc='utf8')

Bases: synapse.lib.output.OutPut

class synapse.lib.output.OutPutStr

Bases: synapse.lib.output.OutPut

synapse.lib.parser module

class synapse.lib.parser.AstConverter(text)

Bases: lark.visitors.Transformer

Convert AST from parser into synapse AST, depth first.

If a method with a name that matches the current rule exists, that will be called, otherwise __default__ will be used

argvquery(kids, meta)

baresubquery(kids, meta)

catchblock(kids, meta)

cmdrargs(kids)

embedquery(kids, meta)

emit(kids, meta)

evalvalu(kids, meta)

exprdict(kids, meta)

exprlist(kids, meta)

funcargs(kids, meta)

A list of function parameters (as part of a function definition)

funccall(kids, meta)

lookup(kids, meta)

operrelprop_join(kids)

operrelprop_pivot(kids, isjoin=False)

query(kids, meta)

stop(kids, meta)

stormcmdargs(kids)

subquery(kids, meta)

switchcase(kids)

tagname(kids)

trycatch(kids, meta)

varderef(kids)

varlist(kids)

yieldvalu(kids)

class synapse.lib.parser.CmdStringer(visit_tokens=True)

Bases: lark.visitors.Transformer

alist(kids, meta)

cmdstring(kids, meta)

valu(kids, meta)

class synapse.lib.parser.Parser(text, offs=0)

Bases: object

Storm query parser

cmdrargs()

Parse command args that might have storm queries as arguments

eval()

lookup()

query()

Parse the storm query

Returns (s_ast.Query): instance of parsed query

synapse.lib.parser.massage_vartokn(x)

synapse.lib.parser.parseEval(text)

synapse.lib.parser.parseQuery(text, mode='storm')

Parse a storm query and return the Lark AST. Cached here to speed up unit tests

synapse.lib.parser.parse_cmd_string(text, off)

Parse a command line string which may be quoted.

synapse.lib.parser.unescape(valu)

Parse a string for backslash-escaped characters and omit them. The full list of escaped characters can be found at https://docs.python.org/3/reference/lexical_analysis.html#string-and-bytes-literals

synapse.lib.provenance module

class synapse.lib.provenance.ProvStor

Bases: synapse.lib.base.Base

Persistently stores provstacks so they can be retrieved by iden and sequence

PROV_FN = 'prov.lmdb'

PROV_MAP_SIZE = 67108864

getProvStack(iden: bytes)

Returns the provenance stack given the iden to it

precommit()

Determine the iden for the current provenance stack and return it

Returns the iden

provStacks(offs, size)

Returns a stream of provenance stacks at the given offset

stor()

Writes the current provenance stack to storage if it wasn’t already there

Returns (iden, provstack) if was written, (None, None) if it was already there

synapse.lib.provenance.claim(typ, **info)

Add an entry to the provenance stack for the duration of the context

synapse.lib.provenance.dupstack(newtask)

Duplicate the current provenance stack onto another task

synapse.lib.provenance.get()

Returns

A tuple of (stack iden (or None if not set), the current provenance stack)

synapse.lib.provenance.reset()

Reset the stack to its initial state

For testing purposes

synapse.lib.provenance.setiden(iden, waswritten)

Sets the cached stack iden, waswritten for the current provenance stack. We use waswritten to cache whether we’ve written the stack and so we can tell the snap whether to fire a prov:new event

synapse.lib.queue module

class synapse.lib.queue.AQueue

Bases: synapse.lib.base.Base

An async queue with chunk optimized sync compatible consumer.

put(item)

Add an item to the queue.

async slice()

class synapse.lib.queue.Queue(maxsize=None)

Bases: object

An asyncio Queue with batch methods and graceful close.

async close()

async put(item)

async puts(items)

async size()

async slice(size=1000)

async slices(size=1000)

class synapse.lib.queue.Window

Bases: synapse.lib.base.Base

A Queue like object which yields added items. If the queue ever reaches its maxsize, it will be fini()d. On fini(), the Window will continue to yield results until empty and then return.

async put(item)

Add a single item to the Window.

async puts(items)

Add multiple items to the window.

synapse.lib.ratelimit module

class synapse.lib.ratelimit.RateLimit(rate, per)

Bases: object

A RateLimit class may be used to detect/enforce rate limits.

Example

# allow 20 uses per 10 sec ( 2/sec ) rlimit = RateLimit(20,10)

Notes

It is best ( even in a “calls per day” type config ) to specify a smaller “per” to force rate “smoothing”.

allows()

Returns True if the rate limit has not been reached.

Example

if not rlimit.allows():

rasie RateExceeded()

# ok to go…

synapse.lib.reflect module

synapse.lib.reflect.getClsNames(item)

Return a list of “fully qualified” class names for an instance.

Example

for name in getClsNames(foo):

print(name)

synapse.lib.reflect.getItemLocals(item)

Iterate the locals of an item and yield (name,valu) pairs.

Example

for name,valu in getItemLocals(item):

dostuff()

synapse.lib.reflect.getMethName(meth)

Return a fully qualified string for the <mod>.<class>.<func> name of a given method.

synapse.lib.reflect.getShareInfo(item)

Get a dictionary of special annotations for a Telepath Proxy.

Parameters

item – Item to inspect.

Notes

This will set the _syn_telemeth attribute on the item and the items class, so this data is only computed once.

Returns

A dictionary of methods requiring special handling by the proxy.

Return type

dict

synapse.lib.rstorm module

class synapse.lib.rstorm.OutPutRst

Bases: synapse.lib.output.OutPutStr

Rst specific helper for output intended to be indented in RST text as a literal block.

prefix = '    '

printf(mesg, addnl=True)

class synapse.lib.rstorm.StormCliOutput

Bases: synapse.tools.storm.StormCli

async handleErr(mesg)

printf(mesg, addnl=True, color=None)

async runRstCmdLine(text, ctx, stormopts=None)

class synapse.lib.rstorm.StormOutput(core, ctx, stormopts=None, opts=None)

Bases: synapse.cmds.cortex.StormCmd

Produce standard output from a stream of storm runtime messages. Must be instantiated for a single query with a rstorm context.

printf(mesg, addnl=True, color=None)

async runCmdLine(line)

Run a line of command input for this command.

Parameters

line (str) – Line to execute

Examples

Run the foo command with some arguments:

await foo.runCmdLine(‘foo –opt baz woot.com’)

async runCmdOpts(opts)

Perform the command actions. Must be implemented by Cmd implementers.

Parameters

opts (dict) – Options dictionary.

class synapse.lib.rstorm.StormRst

Bases: synapse.lib.base.Base

async run()

Parses the specified RST file with Storm directive handling.

Returns

List of line strings for the RST output

Return type

list

synapse.lib.rstorm.getCell(ctor, conf)

synapse.lib.scope module

class synapse.lib.scope.Scope(*frames, **vals)

Bases: object

The Scope object assists in creating nested varible scopes.

Example

with Scope() as scope:

scope.set(‘foo’,10)

with scope:

scope.set(‘foo’,20) dostuff(scope) # ‘foo’ is 20…

dostuff(scope) # ‘foo’ is 10 again…

add(name, *vals)

Add values as iter() compatible items in the current scope frame.

ctor(name, func, *args, **kwargs)

Add a constructor to be called when a specific property is not present.

Example

scope.ctor(‘foo’,FooThing) … foo = scope.get(‘foo’)

enter(vals=None)

Add an additional scope frame.

get(name, defval=None)

Retrieve a value from the closest scope frame.

iter(name)

Iterate through values added with add() from each scope frame.

leave()

Pop the current scope frame.

pop(name, defval=None)

Pop and return a value (from the last frame) of the scope.

Parameters

name (str) – The name of the scope variable.

Returns

The scope variable value or None

Return type

obj

set(name, valu)

Set a value in the current scope frame.

update(vals)

Set multiple values in the current scope frame.

synapse.lib.scope.ctor(name, func, *args, **kwargs)

Add a ctor callback to the global scope.

synapse.lib.scope.enter(vals=None)

Return the task’s local scope for use in a with block

synapse.lib.scope.get(name, defval=None)

Access this task’s scope with default values from glob.

synapse.lib.scope.pop(name)

Pop and return a task scope variable. :param name: The task scope variable name. :type name: str

Returns

The scope value or None

Return type

obj

synapse.lib.scope.set(name, valu)

Set a value in the current frame of the local task scope.

synapse.lib.scope.update(vals)

synapse.lib.scrape module

synapse.lib.scrape.contextScrape(text, form=None, refang=True, first=False)

Scrape types from a blob of text and yield info dictionaries.

Parameters

• text (str) – Text to scrape.

• ptype (str) – Optional ptype to scrape. If present, only scrape items which match the provided type.

• refang (bool) – Whether to remove de-fanging schemes from text before scraping.

• first (bool) – If true, only yield the first item scraped.

Notes

The dictionaries yielded by this function contains the following keys:

match

The raw matching text found in the input text.

offset

The offset into the text where the match was found.

valu

The resulting value.

form

The corresponding form for the valu.

Returns

Yield info dicts of results.

Return type

(dict)

synapse.lib.scrape.cve_check(match: _regex.Match)

synapse.lib.scrape.fqdn_check(match: _regex.Match)

synapse.lib.scrape.fqdn_prefix_check(match: _regex.Match)

synapse.lib.scrape.genFangRegex(fangs, flags=RegexFlag.I)

synapse.lib.scrape.genMatches(text: str, regx: regex.regex.compile, opts: dict)

Generate regular expression matches for a blob of text.

Parameters

• text (str) – The text to generate matches for.

• regx (regex.Regex) – A compiled regex object. The regex must contained a named match group for valu.

• opts (dict) – An options dictionary.

Notes

The dictionaries yielded by this function contains the following keys:

raw_valu

The raw matching text found in the input text.

offset

The offset into the text where the match was found.

valu

The resulting value - this may be altered by callbacks.

The options dictionary can contain a callback key. This function is expected to take a single argument, a regex.Match object, and return a tuple of the new valu and info dictionary. The new valu is used as the valu key in the returned dictionary, and any other information in the info dictionary is pushed into the return dictionary as well.

Yields

dict – A dictionary of match results.

synapse.lib.scrape.getForms()

Get a list of forms recognized by the scrape APIs.

Returns

A list of form values.

Return type

list

synapse.lib.scrape.refang_text(txt)

Remove address de-fanging in text blobs, .e.g. example[.]com to example.com

Matches to keys in FANGS is case-insensitive, but replacement will always be with the lowercase version of the re-fanged value. For example, HXXP://FOO.COM will be returned as http://FOO.COM

Returns

Re-fanged text blob

Return type

(str)

synapse.lib.scrape.refang_text2(txt: str, re: regex.regex.compile = regex.Regex('hxxp:|hxxps:|hxxp\\[:\\]|hxxps\\[:\\]|hxxp\\[://\\]|hxxps\\[://\\]|hxxp\\(:\\)|hxxps\\(:\\)|\\[\\.\\]|\\[．\\]|\\[。\\]|\\[｡\\]|\\(\\.\\)|\\(．\\)|\\(。\\)|\\(｡\\)|\\[:\\]|fxp|fxps|\\[at\\]|\\[@\\]|\\\\\\.', flags=regex.I | regex.V0), fangs: dict = {'(.)': '.', '(。)': '。', '(．)': '．', '(｡)': '｡', '[.]': '.', '[:]': ':', '[@]': '@', '[at]': '@', '[。]': '。', '[．]': '．', '[｡]': '｡', '\\.': '.', 'fxp': 'ftp', 'fxps': 'ftps', 'hxxp(:)': 'http:', 'hxxp:': 'http:', 'hxxp[://]': 'http://', 'hxxp[:]': 'http:', 'hxxps(:)': 'https:', 'hxxps:': 'https:', 'hxxps[://]': 'https://', 'hxxps[:]': 'https:'})

Remove address de-fanging in text blobs, .e.g. example[.]com to example.com

Notes

Matches to keys in FANGS is case-insensitive, but replacement will always be with the lowercase version of the re-fanged value. For example, HXXP://FOO.COM will be returned as http://FOO.COM

Parameters

txt (str) – The text to re-fang.

Returns

A tuple containing the new text, and a dictionary containing offset information where the new text was altered with respect to the original text.

Return type

tuple(str, dict)

synapse.lib.scrape.scrape(text, ptype=None, refang=True, first=False)

Scrape types from a blob of text and return node tuples.

Parameters

• text (str) – Text to scrape.

• ptype (str) – Optional ptype to scrape. If present, only scrape items which match the provided type.

• refang (bool) – Whether to remove de-fanging schemes from text before scraping.

• first (bool) – If true, only yield the first item scraped.

Returns

Yield tuples of node ndef values.

Return type

(str, object)

synapse.lib.share module

class synapse.lib.share.Share

Bases: synapse.lib.base.Base

Class to wrap a dynamically shared object.

synapse.lib.slaboffs module

class synapse.lib.slaboffs.SlabOffs(slab: synapse.lib.lmdbslab.Slab, db: str)

Bases: object

A helper for storing offset integers by iden.

As with all slab objects, this is meant for single-thread async loop use.

delete(iden)

get(iden)

set(iden, offs)

synapse.lib.slabseqn module

class synapse.lib.slabseqn.SlabSeqn(slab, name: str)

Bases: object

An append optimized sequence of byte blobs.

Parameters

• lenv (lmdb.Environment) – The LMDB Environment.

• name (str) – The name of the sequence.

add(item, indx=None)

Add a single item to the sequence.

async aiter(offs, wait=False, timeout=None)

Iterate over items in a sequence from a given offset.

Parameters

• offs (int) – The offset to begin iterating from.

• wait (boolean) – Once caught up, yield new results in realtime.

• timeout (int) – Max time to wait for a new item.

Yields

(indx, valu) – The index and valu of the item.

async cull(offs)

Remove entries up to (and including) the given offset.

first()

get(offs)

Retrieve a single row by offset

getByIndxByts(indxbyts)

getOffsetEvent(offs)

Returns an asyncio Event that will be set when the particular offset is written. The event will be set if the offset has already been reached.

getraw(byts)

async gets(offs, wait=True)

Returns an async generator of indx/valu tuples, optionally waiting and continuing to yield them as new entries are added

Parameters

• offs (int) – The offset to begin iterating from.

• wait (bool) – Whether to continue yielding tupls when it hits the end of the sequence.

Yields

(indx, valu) – The index and valu of the item.

index()

Return the current index to be used

iter(offs)

Iterate over items in a sequence from a given offset.

Parameters

offs (int) – The offset to begin iterating from.

Yields

(indx, valu) – The index and valu of the item.

iterBack(offs)

Iterate backwards over items in a sequence from a given offset.

Parameters

offs (int) – The offset to begin iterating from.

Yields

(indx, valu) – The index and valu of the item.

last()

nextindx()

Determine the next insert offset according to storage.

Returns

The next insert offset.

Return type

int

pop(offs)

Pop a single entry at the given offset.

rows(offs)

Iterate over raw indx, bytes tuples from a given offset.

save(items)

Save a series of items to a sequence.

Parameters

items (tuple) – The series of items to save into the sequence.

Returns

The index of the first item

slice(offs, size)

sliceBack(offs, size)

stat()

trim(offs)

Delete entries starting at offset and moving forward.

async waitForOffset(offs, timeout=None)

Returns

true if the event got set, False if timed out

synapse.lib.snap module

class synapse.lib.snap.ProtoNode(ctx, buid, form, valu, node)

Bases: object

A prototype node used for staging node adds using a SnapEditor.

TODO: This could eventually fully mirror the synapse.lib.node.Node API and be used

to slipstream into sections of the pipeline to facilitate a bulk edit / transaction

async addEdge(verb, n2iden)

async addTag(tag, valu=(None, None), tagnode=None)

get(name)

async getData(name)

getNodeEdit()

getTag(tag)

getTagProp(tag, name)

iden()

async set(name, valu, norminfo=None)

async setData(name, valu)

async setTagProp(tag, name, valu)

class synapse.lib.snap.Scrubber(rules)

Bases: object

scrub(pode)

class synapse.lib.snap.Snap

Bases: synapse.lib.base.Base

A “snapshot” is a transaction across multiple Cortex layers.

The Snap object contains the bulk of the Cortex API to facilitate performance through careful use of transaction boundaries.

Transactions produce the following EventBus events:

(…any splice…) (‘log’, {‘level’: ‘mesg’: }) (‘print’, {}),

async addFeedData(name, items)

async addFeedNodes(name, items)

Call a feed function and return what it returns (typically yields Node()s).

Parameters

• name (str) – The name of the feed record type.

• items (list) – A list of records of the given feed type.

Returns

The return value from the feed function. Typically Node() generator.

Return type

(object)

async addNode(name, valu, props=None, norminfo=None)

Add a node by form name and value with optional props.

Parameters

• name (str) – The form of node to add.

• valu (obj) – The value for the node.

• props (dict) – Optional secondary properties for the node.

Notes

If a props dictionary is provided, it may be mutated during node construction.

Returns

A Node object. It may return None if the snap is unable to add or lift the node.

Return type

s_node.Node

async addNodes(nodedefs)

Add/merge nodes in bulk.

The addNodes API is designed for bulk adds which will also set properties, add tags, add edges, and set nodedata to existing nodes. Nodes are specified as a list of the following tuples:

( (form, valu), {‘props’:{}, ‘tags’:{}})

Parameters

nodedefs (list) – A list of nodedef tuples.

Returns

A list of xact messages.

Return type

(list)

async addStormRuntime(query, opts=None, user=None)

async applyNodeEdit(edit)

async applyNodeEdits(edits)

Sends edits to the write layer and evaluates the consequences (triggers, node object updates)

buidcachesize = 100000

async clearCache()

disableTriggers()

eval(text, opts=None, user=None)

Run a storm query and yield Node() objects.

getEditor()

async getNodeByBuid(buid)

Retrieve a node tuple by binary id.

Parameters

buid (bytes) – The binary ID for the node.

Returns

The node object or None.

Return type

Optional[s_node.Node]

async getNodeByNdef(ndef)

Return a single Node by (form,valu) tuple.

Parameters

• ndef ((str,obj)) – A (form,valu) ndef tuple. valu must be

• normalized. –

Returns

The Node or None.

Return type

(synapse.lib.node.Node)

async getNodeData(buid, name, defv=None)

Get nodedata from closest to write layer, no merging involved

getNodeEditor(node)

async getRuntNodes(full, valu=None, cmpr=None)

async getSnapMeta()

Retrieve snap metadata to store along side nodeEdits.

getStormRuntime(query, opts=None, user=None)

async getTagNode(name)

Retrieve a cached tag node. Requires name is normed. Does not add.

async getTagNorm(tagname)

async hasNodeData(buid, name)

Return True if the buid has nodedata set on it under the given name False otherwise

async hasNodeEdge(buid1, verb, buid2)

async iterNodeData(buid)

Returns: Iterable[Tuple[str, Any]]

async iterNodeEdgesN1(buid, verb=None)

async iterNodeEdgesN2(buid, verb=None)

async iterStormPodes(text, opts, user=None)

Yield packed node tuples for the given storm query text.

async nodes(text, opts=None, user=None)

async nodesByDataName(name)

async nodesByProp(full)

async nodesByPropArray(full, cmpr, valu)

async nodesByPropTypeValu(name, valu)

async nodesByPropValu(full, cmpr, valu)

async nodesByTag(tag, form=None)

async nodesByTagProp(form, tag, name)

async nodesByTagPropValu(form, tag, name, cmpr, valu)

async nodesByTagValu(tag, cmpr, valu, form=None)

async printf(mesg)

async saveNodeEdits(edits, meta)

storm(text, opts=None, user=None)

Execute a storm query and yield (Node(), Path()) tuples.

tagcachesize = 1000

async warn(mesg, log=True, **info)

async warnonce(mesg, log=True, **info)

class synapse.lib.snap.SnapEditor(snap)

Bases: object

A SnapEditor allows tracking node edits with subs/deps as a transaction.

async addNode(formname, valu, props=None, norminfo=None)

getNodeEdits()

loadNode(node)

synapse.lib.spooled module

class synapse.lib.spooled.Dict

Bases: synapse.lib.spooled.Spooled

get(key, defv=None)

has(key)

items()

keys()

async set(key, val)

class synapse.lib.spooled.Set

Bases: synapse.lib.spooled.Spooled

A minimal set-like implementation that will spool to a slab on large growth.

async add(valu)

discard(valu)

class synapse.lib.spooled.Spooled

Bases: synapse.lib.base.Base

A Base class that can be used to implement objects which fallback to lmdb.

These objects are intended to fallback from Python to lmbd slabs, which aligns them together. Under memory pressure, these objects have a better shot of getting paged out.

synapse.lib.storm module

class synapse.lib.storm.BackgroundCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Execute a query pipeline as a background task. NOTE: Variables are passed through but nodes are not

async execStormCmd(runt, genr)

Abstract base method

async execStormTask(query, opts)

getArgParser()

name = 'background'

class synapse.lib.storm.Cmd(runt, runtsafe)

Bases: object

A one line description of the command.

Command usage details and long form description.

Example

cmd –help

Notes

Python Cmd implementers may override the forms attribute with a dictionary to provide information about Synapse forms which are possible input and output nodes that a Cmd may recognize. A list of (key, form) tuples may also be added to provide information about forms which may have additional nodedata added to them by the Cmd.

Example:

{
    'input': (
        'inet:ipv4',
        'tel:mob:telem',
    ),
    'output': (
        'geo:place',
    ),
    'nodedata': (
        ('foodata', 'inet:http:request'),
        ('bardata', 'inet:ipv4'),
    ),
}

asroot = False

async execStormCmd(runt, genr)

Abstract base method

forms = {}

getArgParser()

classmethod getCmdBrief()

getDescr()

getName()

classmethod getStorNode(form)

isReadOnly()

name = 'cmd'

pkgname = ''

readonly = False

async setArgv(argv)

svciden = ''

class synapse.lib.storm.CountCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Iterate through query results, and print the resulting number of nodes which were lifted. This does yield the nodes counted.

Example

foo:bar:size=20 | count

async execStormCmd(runt, genr)

Abstract base method

name = 'count'

readonly = True

class synapse.lib.storm.DelNodeCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Delete nodes produced by the previous query logic.

(no nodes are returned)

Example

inet:fqdn=vertex.link | delnode

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'delnode'

class synapse.lib.storm.DiffCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Generate a list of nodes with changes in the top layer of the current view.

async execStormCmd(runt, genr)

Abstract base method

name = 'diff'

class synapse.lib.storm.DivertCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Either consume a generator or yield it’s results based on a conditional.

NOTE: This command is purpose built to facilitate the –yield convention

common to storm commands.

NOTE: The genr argument must not be a function that returns, else it will

be invoked for each inbound node.

Example

divert $cmdopts.yield $fooBarBaz()

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'divert'

class synapse.lib.storm.DmonManager

Bases: synapse.lib.base.Base

Manager for StormDmon objects.

async addDmon(iden, ddef)

getDmon(iden)

getDmonDef(iden)

getDmonDefs()

getDmonRunlog(iden)

async popDmon(iden)

Remove the dmon and fini it if its exists.

async start()

Start all the dmons.

async stop()

Stop all the dmons.

class synapse.lib.storm.EdgesDelCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Bulk delete light edges from input nodes.

Examples

# Delete all “foo” light edges from an inet:ipv4 inet:ipv4=1.2.3.4 | edges.del foo

# Delete light edges with any verb from a node inet:ipv4=1.2.3.4 | edges.del *

# Delete all “foo” light edges to an inet:ipv4 inet:ipv4=1.2.3.4 | edges.del foo –n2

async delEdges(node, verb, n2=False)

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'edges.del'

class synapse.lib.storm.GraphCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Generate a subgraph from the given input nodes and command line options.

Example

Using the graph command:

inet:fqdn | graph
            --degrees 2
            --filter { -#nope }
            --pivot { <- meta:seen <- meta:source }
            --form-pivot inet:fqdn {<- * | limit 20}
            --form-pivot inet:fqdn {-> * | limit 20}
            --form-filter inet:fqdn {-inet:fqdn:issuffix=1}
            --form-pivot syn:tag {-> *}
            --form-pivot * {-> #}

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'graph'

class synapse.lib.storm.HelpCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

List available commands and a brief description for each.

Examples

// Get all available commands and their brief descriptions.

help

// Only get commands which have “model” in the name.

help model

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'help'

class synapse.lib.storm.IdenCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Lift nodes by iden.

Example

iden b25bc9eec7e159dce879f9ec85fb791f83b505ac55b346fcb64c3c51e98d1175 | count

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'iden'

readonly = True

class synapse.lib.storm.IntersectCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Yield an intersection of the results of running inbound nodes through a pivot.

Note

This command must consume the entire inbound stream to produce the intersection. This type of stream consuming before yielding results can cause the query to appear laggy in comparison with normal incremental stream operations.

Examples

// Show the it:mitre:attack:technique nodes common to several groups

it:mitre:attack:group*in=(G0006, G0007) | intersect { -> it:mitre:attack:technique }

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'intersect'

class synapse.lib.storm.LiftByVerb(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Lift nodes from the current view by an light edge verb.

Examples

# Lift all the n1 nodes for the light edge “foo” lift.byverb “foo”

# Lift all the n2 nodes for the light edge “foo” lift.byverb –n2 “foo”

Notes

Only a single instance of a node will be yielded from this command when that node is lifted via the light edge membership.

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

async iterEdgeNodes(verb, idenset, n2=False)

name = 'lift.byverb'

class synapse.lib.storm.LimitCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Limit the number of nodes generated by the query in the given position.

Example

inet:ipv4 | limit 10

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'limit'

readonly = True

class synapse.lib.storm.MaxCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Consume nodes and yield only the one node with the highest value for a property or variable.

Examples

file:bytes +#foo.bar | max :size

file:bytes +#foo.bar +.seen ($tick, $tock) = .seen | max $tick

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'max'

readonly = True

class synapse.lib.storm.MergeCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Merge edits from the incoming nodes down to the next layer.

NOTE: This command requires the current view to be a fork.

NOTE: The arguments for including/excluding tags can accept tag glob

expressions for specifying tags. For more information on tag glob expressions, check the Synapse documentation for $node.globtags().

Examples

// Having tagged a new #cno.mal.redtree subgraph in a forked view…

#cno.mal.redtree | merge –apply

// Print out what the merge command would do but dont.

#cno.mal.redtree | merge

// Merge ou:org nodes, but when merging tags, only merge tags one level // below the rep.vt and rep.whoxy tags.

ou:org | merge –include-tags rep.vt.* rep.whoxy.* –apply

// Only merge tags, and exclude any tags in the cno tag tree.

ou:org | merge –only-tags –exclude-tags cno.** –apply

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'merge'

class synapse.lib.storm.MinCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Consume nodes and yield only the one node with the lowest value for a property.

Examples

file:bytes +#foo.bar | min :size

file:bytes +#foo.bar | min .seen

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'min'

readonly = True

class synapse.lib.storm.MoveTagCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Rename an entire tag tree and preserve time intervals.

Example

movetag foo.bar baz.faz.bar

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'movetag'

class synapse.lib.storm.OnceCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

The once command ensures that a node makes it through the once command but a single time, even across independent queries. The gating is keyed by a required name parameter to the once command, so a node can be run through different queries, each a single time, so long as the names differ.

For example, to run an enrichment command on a set of nodes just once:

file:bytes#my.files | once enrich:foo | enrich.foo

If you insert the once command with the same name on the same nodes, they will be dropped from the pipeline. So in the above example, if we run it again, the enrichment will not run a second time, as all the nodes will be dropped from the pipeline before reaching the enrich.foo portion of the pipeline.

Simlarly, running this:

file:bytes#my.files | once enrich:foo

Also yields no nodes. And even though the rest of the pipeline is different, this query:

file:bytes#my.files | once enrich:foo | enrich.bar

would not run the enrich.bar command, as the name “enrich:foo” has already been seen to occur on the file:bytes passing through the once command, so all of the nodes will be dropped from the pipeline.

However, this query:

file:bytes#my.files | once look:at:my:nodes

Would yield all the file:bytes tagged with #my.files, as the name parameter given to the once command differs from the original “enrich:foo”.

The once command utilizes a node’s nodedata cache, and you can use the –asof parameter to update the named action’s timestamp in order to bypass/update the once timestamp. So this command:

inet:ipv4#my.addresses | once node:enrich –asof now | my.enrich.command

Will yield all the enriched nodes the first time around. The second time that command is run, all of those nodes will be re-enriched, as the asof timestamp will be greater the second time around, so no nodes will be dropped.

As state tracking data for the once command is stored as nodedata, it is stored in your view’s write layer, making it view-specific. So if you have two views, A and B, and they do not share any layers between them, and you execute this query in view A:

inet:ipv4=8.8.8.8 | once enrich:address | enrich.baz

And then you run it in view B, the node will still pass through the once command to the enrich.baz portion of the pipeline, as the nodedata for the once command does not yet exist in view B.

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'once'

class synapse.lib.storm.ParallelCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Execute part of a query pipeline in parallel. This can be useful to minimize round-trip delay during enrichments.

Examples

inet:ipv4#foo | parallel { $place = $lib.import(foobar).lookup(:latlong) [ :place=$place ] }

NOTE: Storm variables set within the parallel query pipelines do not interact.

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'parallel'

async nextitem(inq)

async pipeline(runt, query, inq, outq)

readonly = True

class synapse.lib.storm.Parser(prog=None, descr=None, root=None)

Bases: object

add_argument(*names, **opts)

help(mesg=None)

parse_args(argv)

set_inputs(idefs)

class synapse.lib.storm.PureCmd(cdef, runt, runtsafe)

Bases: synapse.lib.storm.Cmd

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

getDescr()

getName()

readonly = True

class synapse.lib.storm.ReIndexCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Use admin privileges to re index/normalize node properties.

NOTE: Currently does nothing but is reserved for future use.

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'reindex'

class synapse.lib.storm.RunAsCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Execute a storm query as a specified user.

NOTE: This command requires admin privileges.

Examples

// Create a node as another user. runas someuser { [ inet:fqdn=foo.com ] }

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'runas'

class synapse.lib.storm.Runtime

Bases: synapse.lib.base.Base

A Runtime represents the instance of a running query.

The runtime should maintain a firm API boundary using the snap. Parallel query execution requires that the snap be treated as an opaque object which is called through, but not dereferenced.

addInput(node)

Add a Node() object as input to the query runtime.

allowed(perms, gateiden=None, default=False)

cancel()

confirm(perms, gateiden=None, default=False)

Raise AuthDeny if user doesn’t have global permissions and write layer permissions

async coreDynCall(todo, perm=None)

async dyncall(iden, todo, gatekeys=())

async dyniter(iden, todo, gatekeys=())

async emit(item)

async emitter()

async execute(genr=None)

getCmdRuntime(query, opts=None)

Yield a runtime with proper scoping for use in executing a pure storm command.

async getInput()

async getModRuntime(query, opts=None)

Construct a non-context managed runtime for use in module imports.

async getOneNode(propname, valu, filt=None, cmpr='=')

Return exactly 1 node by <prop> <cmpr> <valu>

getOpt(name, defval=None)

async getStormQuery(text)

getSubRuntime(query, opts=None)

Yield a runtime with shared scope that will populate changes upward.

async getTeleProxy(url, **opts)

getVar(name, defv=None)

initPath(node)

async initSubRuntime(query, opts=None)

Construct and return sub-runtime with a shared scope. ( caller must fini )

isAdmin(gateiden=None)

isRuntVar(name)

layerConfirm(perms)

async popVar(name)

async printf(mesg)

async reqGateKeys(gatekeys)

async reqUserCanReadLayer(layriden)

setOpt(name, valu)

async setVar(name, valu)

async storm(text, opts=None, genr=None)

Execute a storm runtime which inherits from this storm runtime.

tick()

async warn(mesg, **info)

async warnonce(mesg, **info)

class synapse.lib.storm.ScrapeCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Use textual properties of existing nodes to find other easily recognizable nodes.

Examples

# Scrape properties from inbound nodes and create standalone nodes. inet:search:query | scrape

# Scrape properties from inbound nodes and make refs light edges to the scraped nodes. inet:search:query | scrape –refs

# Scrape only the :engine and :text props from the inbound nodes. inet:search:query | scrape :text :engine

# Scrape properties inbound nodes and yield newly scraped nodes. inet:search:query | scrape –yield

# Skip re-fanging text before scraping. inet:search:query | scrape –skiprefang

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'scrape'

class synapse.lib.storm.SleepCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Introduce a delay between returning each result for the storm query.

NOTE: This is mostly used for testing / debugging.

Example

#foo.bar | sleep 0.5

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'sleep'

readonly = True

class synapse.lib.storm.SpinCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Iterate through all query results, but do not yield any. This can be used to operate on many nodes without returning any.

Example

foo:bar:size=20 [ +#hehe ] | spin

async execStormCmd(runt, genr)

Abstract base method

name = 'spin'

readonly = True

class synapse.lib.storm.SpliceListCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Retrieve a list of splices backwards from the end of the splicelog.

Examples

# Show the last 10 splices. splice.list | limit 10

# Show splices after a specific time. splice.list –mintime “2020/01/06 15:38:10.991”

# Show splices from a specific timeframe. splice.list –mintimestamp 1578422719360 –maxtimestamp 1578422719367

Notes

If both a time string and timestamp value are provided for a min or max, the timestamp will take precedence over the time string value.

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'splice.list'

readonly = True

class synapse.lib.storm.SpliceUndoCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Reverse the actions of syn:splice runt nodes.

Examples

# Undo the last 5 splices. splice.list | limit 5 | splice.undo

# Undo splices after a specific time. splice.list –mintime “2020/01/06 15:38:10.991” | splice.undo

# Undo splices from a specific timeframe. splice.list –mintimestamp 1578422719360 –maxtimestamp 1578422719367 | splice.undo

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'splice.undo'

async undoNodeAdd(runt, splice, node)

async undoNodeDel(runt, splice, node)

async undoPropDel(runt, splice, node)

async undoPropSet(runt, splice, node)

async undoTagAdd(runt, splice, node)

async undoTagDel(runt, splice, node)

async undoTagPropDel(runt, splice, node)

async undoTagPropSet(runt, splice, node)

class synapse.lib.storm.StormDmon

Bases: synapse.lib.base.Base

A background storm runtime which is restarted by the cortex.

async bump()

async dmonloop()

pack()

async run()

async stop()

class synapse.lib.storm.SudoCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Deprecated sudo command.

Left in for 2.x.x so that Storm command with it are still valid to execute.

async execStormCmd(runt, genr)

Abstract base method

name = 'sudo'

class synapse.lib.storm.TagPruneCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Prune a tag (or tags) from nodes.

This command will delete the tags specified as parameters from incoming nodes, as well as all of their parent tags that don’t have other tags as children.

For example, given a node with the tags:

#parent #parent.child #parent.child.grandchild

Pruning the parent.child.grandchild tag would remove all tags. If the node had the tags:

#parent #parent.child #parent.child.step #parent.child.grandchild

Pruning the parent.child.grandchild tag will only remove the parent.child.grandchild tag as the parent tags still have other children.

Examples

# Prune the parent.child.grandchild tag inet:ipv4=1.2.3.4 | tag.prune parent.child.grandchild

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

hasChildTags(node, tag)

name = 'tag.prune'

class synapse.lib.storm.TeeCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Execute multiple Storm queries on each node in the input stream, joining output streams together.

Commands are executed in order they are given; unless the --parallel switch is provided.

Examples

# Perform a pivot out and pivot in on a inet:ivp4 node inet:ipv4=1.2.3.4 | tee { -> * } { <- * }

# Also emit the inbound node inet:ipv4=1.2.3.4 | tee –join { -> * } { <- * }

# Execute multiple enrichment queries in parallel. inet:ipv4=1.2.3.4 | tee -p { enrich.foo } { enrich.bar } { enrich.baz }

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'tee'

async pipeline(runt, outq, genr=None)

readonly = True

class synapse.lib.storm.TreeCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Walk elements of a tree using a recursive pivot.

Examples

# pivot upward yielding each FQDN inet:fqdn=www.vertex.link | tree { :domain -> inet:fqdn }

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'tree'

readonly = True

class synapse.lib.storm.UniqCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Filter nodes by their uniq iden values. When this is used a Storm pipeline, only the first instance of a given node is allowed through the pipeline.

Examples

#badstuff +inet:ipv4 ->* | uniq

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'uniq'

readonly = True

class synapse.lib.storm.ViewExecCmd(runt, runtsafe)

Bases: synapse.lib.storm.Cmd

Execute a storm query in a different view.

NOTE: Variables are passed through but nodes are not

Examples

// Move some tagged nodes to another view inet:fqdn#foo.bar $fqdn=$node.value() | view.exec 95d5f31f0fb414d2b00069d3b1ee64c6 { [ inet:fqdn=$fqdn ] }

async execStormCmd(runt, genr)

Abstract base method

getArgParser()

name = 'view.exec'

readonly = True

synapse.lib.storm_format module

class synapse.lib.storm_format.StormLexer(parser)

Bases: pygments.lexer.Lexer

get_tokens_unprocessed(text)

Return an iterable of (index, tokentype, value) pairs where “index” is the starting position of the token within the input text.

In subclasses, implement this method as a generator to maximize effectiveness.

synapse.lib.storm_format.highlight_storm(parser, text)

Prints a storm query with syntax highlighting

synapse.lib.stormctrl module

exception synapse.lib.stormctrl.StormBreak(item=None)

Bases: synapse.lib.stormctrl.StormCtrlFlow

exception synapse.lib.stormctrl.StormContinue(item=None)

Bases: synapse.lib.stormctrl.StormCtrlFlow

exception synapse.lib.stormctrl.StormCtrlFlow(item=None)

Bases: Exception

exception synapse.lib.stormctrl.StormExit(item=None)

Bases: synapse.lib.stormctrl.StormCtrlFlow

exception synapse.lib.stormctrl.StormReturn(item=None)

Bases: synapse.lib.stormctrl.StormCtrlFlow

exception synapse.lib.stormctrl.StormStop(item=None)

Bases: synapse.lib.stormctrl.StormCtrlFlow

synapse.lib.stormhttp module

class synapse.lib.stormhttp.HttpResp(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a HTTP response.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormhttp.LibHttp(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library exposing an HTTP client API.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async inetHttpConnect(url, headers=None, ssl_verify=True, timeout=300)

strify(item)

class synapse.lib.stormhttp.WebSocket

Bases: synapse.lib.base.Base, synapse.lib.stormtypes.StormType

Implements the Storm API for a Websocket.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async rx(timeout=None)

async tx(mesg)

synapse.lib.stormsvc module

class synapse.lib.stormsvc.StormSvc

Bases: object

The StormSvc mixin class used to make a remote storm service with commands.

async getStormSvcInfo()

async getStormSvcPkgs()

class synapse.lib.stormsvc.StormSvcClient

Bases: synapse.lib.base.Base

A StormService is a wrapper for a telepath proxy to a service accessible from the storm runtime.

synapse.lib.stormtypes module

class synapse.lib.stormtypes.Bool(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a boolean instance.

class synapse.lib.stormtypes.Bytes(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a Bytes object.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async slice(start, end=None)

async unpack(fmt, offset=0)

class synapse.lib.stormtypes.CmdOpts(valu, path=None)

Bases: synapse.lib.stormtypes.Dict

A dictionary like object that holds a reference to a command options namespace. ( This allows late-evaluation of command arguments rather than forcing capture )

async deref(name)

async iter()

async setitem(name, valu)

async stormrepr()

async value()

class synapse.lib.stormtypes.CronJob(runt, cdef, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm api for a cronjob instance.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.Dict(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a Dictionary object.

async deref(name)

async iter()

async setitem(name, valu)

async stormrepr()

async value()

class synapse.lib.stormtypes.Gate(runt, valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for an AuthGate.

class synapse.lib.stormtypes.Layer(runt, ldef, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm api for a layer instance.

async getMirrorStatus()

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async getStorNode(nodeid)

async getStorNodes()

async liftByProp(propname, propvalu=None, propcmpr='=')

async liftByTag(tagname, formname=None)

async verify(config=None)

class synapse.lib.stormtypes.Lib(runt, name=())

Bases: synapse.lib.stormtypes.StormType

A collection of storm methods under a name

addLibFuncs()

async deref(name)

async dyncall(iden, todo, gatekeys=())

async dyniter(iden, todo, gatekeys=())

async stormrepr()

class synapse.lib.stormtypes.LibAuth(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Auth in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

static ruleFromText(text)

class synapse.lib.stormtypes.LibAxon(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm library for interacting with the Cortex’s Axon.

async del_(sha256)

async dels(sha256s)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async jsonlines(sha256)

async list(offs=0, wait=False, timeout=None)

async readlines(sha256)

strify(item)

async urlfile(*args, **kwargs)

async wget(url, headers=None, params=None, method='GET', json=None, body=None, ssl=True, timeout=None)

async wput(sha256, url, headers=None, params=None, method='PUT', ssl=True, timeout=None)

class synapse.lib.stormtypes.LibBase(runt, name=())

Bases: synapse.lib.stormtypes.Lib

The Base Storm Library. This mainly contains utility functionality.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async trycast(name, valu)

class synapse.lib.stormtypes.LibBase64(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for encoding and decoding base64 data.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibBytes(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with bytes storage.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibCron(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Cron Jobs in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibCsv(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with csvtool.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibDmon(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with StormDmons.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibExport(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for exporting data.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async toaxon(query, opts=None)

class synapse.lib.stormtypes.LibFeed(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Cortex feed functions.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibGates(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Auth Gates in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibGlobals(runt, name)

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with global variables which are persistent across the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibJsonStor(runt, name=())

Bases: synapse.lib.stormtypes.Lib

Implements cortex JSON storage.

addLibFuncs()

async get(path, prop=None)

async has(path)

async iter(path=None)

async set(path, valu, prop=None)

class synapse.lib.stormtypes.LibLayer(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Layers in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibLift(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with lift helpers.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibPipe(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm library for interacting with non-persistent queues.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibPkg(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Storm Packages.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibPs(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with running tasks on the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibQueue(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with persistent Queues in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibRegx(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm library for searching/matching with regular expressions.

async findall(pattern, text, flags=0)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async matches(pattern, text, flags=0)

async replace(pattern, replace, text, flags=0)

async search(pattern, text, flags=0)

class synapse.lib.stormtypes.LibRoles(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Auth Roles in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibService(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Storm Services.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibStats(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for statistics related functionality.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async tally()

class synapse.lib.stormtypes.LibStr(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with strings.

async concat(*args)

async format(text, **kwargs)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async join(sepr, items)

class synapse.lib.stormtypes.LibTags(runt, name=())

Bases: synapse.lib.stormtypes.Lib

Storm utility functions for tags.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async prefix(names, prefix)

class synapse.lib.stormtypes.LibTelepath(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for making Telepath connections to remote services.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibTime(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with timestamps.

async day(tick)

async dayofmonth(tick)

async dayofweek(tick)

async dayofyear(tick)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async hour(tick)

async minute(tick)

async month(tick)

async monthofyear(tick)

async second(tick)

async toUTC(tick, timezone)

async year(tick)

class synapse.lib.stormtypes.LibTrigger(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Triggers in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibUser(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with data about the current user.

addLibFuncs()

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibUsers(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Auth Users in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibVars(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with runtime variables.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.LibView(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for interacting with Views in the Cortex.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.List(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a List instance.

async extend(valu)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async iter()

async setitem(name, valu)

async slice(start, end=None)

async stormrepr()

async value()

class synapse.lib.stormtypes.Node(node, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm api for a node instance.

getByLayer()

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async getStorNodes()

class synapse.lib.stormtypes.NodeData(node, path=None)

Bases: synapse.lib.stormtypes.Prim

A Storm Primitive representing the NodeData stored for a Node.

async cacheget(name, asof='now')

async cacheset(name, valu)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.NodeProps(node, path=None)

Bases: synapse.lib.stormtypes.Prim

A Storm Primitive representing the properties on a Node.

async get(name)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async iter()

async list()

async set(prop, valu)

async setitem(name, valu)

Set a property on a Node.

Parameters

• prop (str) – The name of the property to set.

• valu – The value being set.

Raises

• s_exc – NoSuchProp: If the property being set is not valid for the node.

• s_exc.BadTypeValu – If the value of the proprerty fails to normalize.

value()

class synapse.lib.stormtypes.Path(node, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for the Path object.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.PathMeta(path)

Bases: synapse.lib.stormtypes.Prim

Put the storm deref/setitem/iter convention on top of path meta information.

async deref(name)

async iter()

async setitem(name, valu)

class synapse.lib.stormtypes.PathVars(path)

Bases: synapse.lib.stormtypes.Prim

Put the storm deref/setitem/iter convention on top of path variables.

async deref(name)

async iter()

async setitem(name, valu)

class synapse.lib.stormtypes.Pipe(runt, size)

Bases: synapse.lib.stormtypes.StormType

A Storm Pipe provides fast ephemeral queues.

async close()

Close the pipe for writing. This will cause the slice()/slices() API to return once drained.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.Prim(valu, path=None)

Bases: synapse.lib.stormtypes.StormType

The base type for all Storm primitive values.

async bool()

async iter()

async nodes()

async stormrepr()

value()

class synapse.lib.stormtypes.Proxy(runt, proxy, path=None)

Bases: synapse.lib.stormtypes.StormType

Implements the Storm API for a Telepath proxy.

These can be created via $lib.telepath.open(). Storm Service objects are also Telepath proxy objects.

Methods called off of these objects are executed like regular Telepath RMI calls.

An example of calling a method which returns data:

$prox = $lib.telepath.open($url)
$result = $prox.doWork($data)
return ( $result )

An example of calling a method which is a generator:

$prox = $lib.telepath.open($url)
for $item in = $prox.genrStuff($data) {
    $doStuff($item)
}

async deref(name)

async stormrepr()

class synapse.lib.stormtypes.ProxyGenrMethod(meth, path=None)

Bases: synapse.lib.stormtypes.StormType

async stormrepr()

class synapse.lib.stormtypes.ProxyMethod(runt, meth, path=None)

Bases: synapse.lib.stormtypes.StormType

async stormrepr()

class synapse.lib.stormtypes.Query(text, varz, runt, path=None)

Bases: synapse.lib.stormtypes.Prim

A storm primitive representing an embedded query.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async iter()

async nodes()

async stormrepr()

class synapse.lib.stormtypes.Queue(runt, name, info)

Bases: synapse.lib.stormtypes.StormType

A StormLib API instance of a named channel in the Cortex multiqueue.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async stormrepr()

class synapse.lib.stormtypes.Role(runt, valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a Role.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async stormrepr()

async value()

class synapse.lib.stormtypes.Service(runt, ssvc)

Bases: synapse.lib.stormtypes.Proxy

async deref(name)

class synapse.lib.stormtypes.Set(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a Set object.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async iter()

async stormrepr()

class synapse.lib.stormtypes.StatTally(path=None)

Bases: synapse.lib.stormtypes.Prim

A tally object.

An example of using it:

$tally = $lib.stats.tally()

$tally.inc(foo)

for $name, $total in $tally {
    $doStuff($name, $total)
}

async get(name)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async inc(name, valu=1)

async iter()

value()

class synapse.lib.stormtypes.StormHiveDict(runt, info)

Bases: synapse.lib.stormtypes.Prim

A Storm Primitive representing a HiveDict.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async iter()

value()

class synapse.lib.stormtypes.StormType(path=None)

Bases: object

The base type for storm runtime value objects.

async deref(name)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

ismutable()

async setitem(name, valu)

class synapse.lib.stormtypes.StormTypesRegistry

Bases: object

addStormLib(path, ctor)

addStormType(path, ctor)

base_undefined_types = ('any', 'int', 'null', 'time', 'prim', 'undef', 'integer', 'storm:lib', 'generator')

delStormLib(path)

delStormType(path)

getLibDocs()

getTypeDocs()

iterLibs()

iterTypes()

known_types = {'boolean', 'bytes', 'dict', 'list', 'set', 'storm:auth:gate', 'storm:auth:role', 'storm:auth:user', 'storm:auth:user:json', 'storm:cmdopts', 'storm:cronjob', 'storm:hive:dict', 'storm:http:resp', 'storm:http:socket', 'storm:imap:server', 'storm:json:schema', 'storm:layer', 'storm:model:form', 'storm:model:property', 'storm:model:tagprop', 'storm:model:type', 'storm:node', 'storm:node:data', 'storm:node:path:meta', 'storm:node:path:vars', 'storm:node:props', 'storm:oauth:v1:client', 'storm:path', 'storm:pipe', 'storm:project', 'storm:project:epic', 'storm:project:epics', 'storm:project:sprint', 'storm:project:sprints', 'storm:project:ticket', 'storm:project:ticket:comment', 'storm:project:ticket:comments', 'storm:project:tickets', 'storm:proxy', 'storm:query', 'storm:queue', 'storm:smtp:message', 'storm:stat:tally', 'storm:stix:bundle', 'storm:text', 'storm:trigger', 'storm:view', 'str', 'xml:element'}

registerLib(ctor)

Decorator to register a StormLib

registerType(ctor)

Decorator to register a StormPrim

rtypes = {}

undefined_types = {'any', 'generator', 'int', 'integer', 'null', 'prim', 'storm:lib', 'time', 'undef'}

class synapse.lib.stormtypes.Str(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a String object.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.Text(valu, path=None)

Bases: synapse.lib.stormtypes.Prim

A mutable text type for simple text construction.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

class synapse.lib.stormtypes.Trigger(runt, tdef)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a Trigger.

async deref(name)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async move(viewiden)

async pack()

async set(name, valu)

class synapse.lib.stormtypes.Undef

Bases: object

class synapse.lib.stormtypes.User(runt, valu, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm API for a User.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async stormrepr()

async value()

class synapse.lib.stormtypes.UserJson(runt, valu)

Bases: synapse.lib.stormtypes.Prim

Implements per-user JSON storage.

async get(path, prop=None)

async has(path)

async iter(path=None)

async set(path, valu, prop=None)

class synapse.lib.stormtypes.View(runt, vdef, path=None)

Bases: synapse.lib.stormtypes.Prim

Implements the Storm api for a View instance.

async addNode(form, valu, props=None)

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

async viewDynCall(todo, perm)

async viewDynIter(todo, perm)

synapse.lib.stormtypes.allowed(perm, gateiden=None)

synapse.lib.stormtypes.confirm(perm, gateiden=None)

synapse.lib.stormtypes.fromprim(valu, path=None, basetypes=True)

synapse.lib.stormtypes.getCallSig(func) → inspect.Signature

Get the callsig of a function, stripping self if present.

synapse.lib.stormtypes.getDoc(obj, errstr)

Helper to get __doc__

synapse.lib.stormtypes.intify(x)

synapse.lib.stormtypes.ismutable(valu)

async synapse.lib.stormtypes.kwarg_format(_text, **kwargs)

Replaces instances curly-braced argument names in text with their values

synapse.lib.stormtypes.ruleFromText(text)

Get a rule tuple from a text string.

Parameters

text (str) – The string to process.

Returns

A tuple containing a bool and a list of permission parts.

Return type

(bool, tuple)

synapse.lib.stormtypes.stormfunc(readonly=False)

async synapse.lib.stormtypes.tobool(valu, noneok=False)

async synapse.lib.stormtypes.tobuidhex(valu, noneok=False)

async synapse.lib.stormtypes.toint(valu, noneok=False)

async synapse.lib.stormtypes.toiter(valu, noneok=False)

async synapse.lib.stormtypes.toprim(valu, path=None)

async synapse.lib.stormtypes.torepr(valu, usestr=False)

async synapse.lib.stormtypes.tostr(valu, noneok=False)

synapse.lib.stormwhois module

class synapse.lib.stormwhois.LibWhois(runt, name=())

Bases: synapse.lib.stormtypes.Lib

A Storm Library for providing a consistent way to generate guids for WHOIS / Registration Data in Storm.

getObjLocals()

Get the default list of key-value pairs which may be added to the object .locls dictionary.

Returns

A key/value pairs.

Return type

dict

synapse.lib.structlog module

class synapse.lib.structlog.JsonFormatter(*args, **kwargs)

Bases: logging.Formatter

format(record: logging.LogRecord)

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

synapse.lib.task module

class synapse.lib.task.Task

Bases: synapse.lib.base.Base

The synapse Task object implements concepts similar to process trees for asyncio.Task instances.

async kill()

pack()

async worker(coro, name='worker')

synapse.lib.task.current()

Return the current synapse task.

async synapse.lib.task.executor(func, *args, **kwargs)

Execute a function in an executor thread.

Parameters

todo ((func,args,kwargs)) – A todo tuple.

synapse.lib.task.loop()

synapse.lib.task.user()

Return the current task user.

synapse.lib.task.username()

Return the current task user name.

synapse.lib.task.vardefault(name, func)

Add a default constructor for a particular task-local variable

All future calls to taskVarGet with the same name will return the result of calling func

synapse.lib.task.varget(name, defval=None, task=None)

Access a task local variable by name

Precondition:

If task is None, this must be called from task context

synapse.lib.task.varinit(task=None)

Initializes (or re-initializes for testing purposes) all of a task’s task-local variables

Precondition:

If task is None, this must be called from task context

synapse.lib.task.varset(name, valu, task=None)

Set a task-local variable

Parameters

task – If task is None, uses current task

Precondition:

If task is None, this must be called from task context

synapse.lib.thishost module

synapse.lib.thishost.get(prop)

Retrieve a property from the hostinfo dictionary.

Example

import synapse.lib.thishost as s_thishost

if s_thishost.get(‘platform’) == ‘windows’:

dostuff()

synapse.lib.thishost.hostaddr(dest='8.8.8.8')

Retrieve the ipv4 address for this host ( optionally as seen from dest ). .. rubric:: Example

addr = s_socket.hostaddr()

synapse.lib.thisplat module

synapse.lib.threads module

synapse.lib.threads.current()

synapse.lib.threads.iden()

synapse.lib.time module

Time related utilities for synapse “epoch millis” time values.

synapse.lib.time.day(tick)

synapse.lib.time.dayofmonth(tick)

synapse.lib.time.dayofweek(tick)

synapse.lib.time.dayofyear(tick)

synapse.lib.time.delta(text)

Parse a simple time delta string and return the delta.

synapse.lib.time.hour(tick)

synapse.lib.time.ival(*times)

synapse.lib.time.minute(tick)

synapse.lib.time.month(tick)

synapse.lib.time.parse(text, base=None, chop=False)

Parse a time string into an epoch millis value.

Parameters

• text (str) – Time string to parse

• base (int or None) – Milliseconds to offset the time from

• chop (bool) – Whether to chop the digit-only string to 17 chars

Returns

Epoch milliseconds

Return type

int

synapse.lib.time.parsetz(text)

Parse timezone from time string, with UTC as the default.

Parameters

text (str) – Time string

Returns

A tuple of text with tz chars removed and base milliseconds to offset time.

Return type

tuple

synapse.lib.time.repr(tick, pack=False)

Return a date string for an epoch-millis timestamp.

Parameters

tick (int) – The timestamp in milliseconds since the epoch.

Returns

A date time string

Return type

(str)

synapse.lib.time.second(tick)

synapse.lib.time.toUTC(tick, fromzone)

synapse.lib.time.wildrange(text)

Parse an interval from a wild card time stamp: 2021/10/31*

synapse.lib.time.year(tick)

synapse.lib.trigger module

class synapse.lib.trigger.Trigger(view, tdef)

Bases: object

async execute(node, vars=None, view=None)

Actually execute the query

get(name)

getStorNode(form)

pack()

async set(name, valu)

Set one of the dynamic elements of the trigger definition.

class synapse.lib.trigger.Triggers(view)

Bases: object

Manages “triggers”, conditions where changes in data result in new storm queries being executed.

Note

These methods should not be called directly under normal circumstances. Use the owning “View” object to ensure that mirrors/clusters members get the same changes.

get(iden)

list()

async load(tdef)

pop(iden)

async runNodeAdd(node, view=None)

async runNodeDel(node, view=None)

async runPropSet(node, prop, oldv, view=None)

async runTagAdd(node, tag, view=None)

async runTagDel(node, tag, view=None)

synapse.lib.trigger.reqValidTdef(conf)

synapse.lib.types module

class synapse.lib.types.Array(modl, name, info, opts)

Bases: synapse.lib.types.Type

isarray = True

postTypeInit()

repr(valu)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

class synapse.lib.types.Bool(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

repr(valu)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 2

class synapse.lib.types.Comp(modl, name, info, opts)

Bases: synapse.lib.types.Type

getCompOffs(name)

If this type is a compound, return the field offset for the given property name or None.

postTypeInit()

repr(valu)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.Data(modl, name, info, opts)

Bases: synapse.lib.types.Type

norm(valu)

Normalize the value for a given type.

Parameters

valu (obj) – The value to normalize.

Returns

The normalized valu, info tuple.

Return type

((obj,dict))

Notes

The info dictionary uses the following key conventions:

subs (dict): The normalized sub-fields as name: valu entries.

postTypeInit()

stortype: int = 13

class synapse.lib.types.Duration(modl, name, info, opts)

Bases: synapse.lib.types.IntBase

postTypeInit()

repr(valu)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 5

class synapse.lib.types.Edge(modl, name, info, opts)

Bases: synapse.lib.types.Type

getCompOffs(name)

If this type is a compound, return the field offset for the given property name or None.

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.FieldHelper(modl, tname, fields)

Bases: collections.defaultdict

Helper for Comp types. Performs Type lookup/creation upon first use.

class synapse.lib.types.Float(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 22

class synapse.lib.types.Guid(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

stortype: int = 10

class synapse.lib.types.Hex(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

stortype: int = 1

class synapse.lib.types.HugeNum(modl, name, info, opts)

Bases: synapse.lib.types.Type

norm(valu)

Normalize the value for a given type.

Parameters

valu (obj) – The value to normalize.

Returns

The normalized valu, info tuple.

Return type

((obj,dict))

Notes

The info dictionary uses the following key conventions:

subs (dict): The normalized sub-fields as name: valu entries.

stortype: int = 23

class synapse.lib.types.Int(modl, name, info, opts)

Bases: synapse.lib.types.IntBase

merge(oldv, newv)

Allow types to “merge” data from two sources based on value precedence.

Parameters

• valu (object) – The current value.

• newv (object) – The updated value.

Returns

The merged value.

Return type

(object)

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

class synapse.lib.types.IntBase(modl, name, info, opts)

Bases: synapse.lib.types.Type

class synapse.lib.types.Ival(modl, name, info, opts)

Bases: synapse.lib.types.Type

An interval, i.e. a range, of times

merge(oldv, newv)

Allow types to “merge” data from two sources based on value precedence.

Parameters

• valu (object) – The current value.

• newv (object) – The updated value.

Returns

The merged value.

Return type

(object)

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 12

class synapse.lib.types.Loc(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stems(valu)

stortype: int = 15

class synapse.lib.types.Ndef(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.NodeProp(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

stortype: int = 13

class synapse.lib.types.Range(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.Str(modl, name, info, opts)

Bases: synapse.lib.types.Type

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 1

class synapse.lib.types.Tag(modl, name, info, opts)

Bases: synapse.lib.types.Str

postTypeInit()

class synapse.lib.types.TagPart(modl, name, info, opts)

Bases: synapse.lib.types.Str

postTypeInit()

class synapse.lib.types.Taxon(modl, name, info, opts)

Bases: synapse.lib.types.Str

postTypeInit()

class synapse.lib.types.Taxonomy(modl, name, info, opts)

Bases: synapse.lib.types.Str

postTypeInit()

class synapse.lib.types.Time(modl, name, info, opts)

Bases: synapse.lib.types.IntBase

getTickTock(vals)

Get a tick, tock time pair.

Parameters

vals (list) – A pair of values to norm.

Returns

A ordered pair of integers.

Return type

(int, int)

merge(oldv, newv)

Allow types to “merge” data from two sources based on value precedence.

Parameters

• valu (object) – The current value.

• newv (object) – The updated value.

Returns

The merged value.

Return type

(object)

postTypeInit()

repr(valu)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 11

class synapse.lib.types.TimeEdge(modl, name, info, opts)

Bases: synapse.lib.types.Edge

getCompOffs(name)

If this type is a compound, return the field offset for the given property name or None.

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

stortype: int = 13

class synapse.lib.types.Type(modl, name, info, opts)

Bases: object

clone(opts)

Create a new instance of this type with the specified options.

Parameters

opts (dict) – The type specific options for the new instance.

cmpr(val1, name, val2)

Compare the two values using the given type specific comparator.

extend(name, opts, info)

Extend this type to construct a sub-type.

Parameters

• name (str) – The name of the new sub-type.

• opts (dict) – The type options for the sub-type.

• info (dict) – The type info for the sub-type.

Returns

A new sub-type instance.

Return type

(synapse.types.Type)

getCmprCtor(name)

getCompOffs(name)

If this type is a compound, return the field offset for the given property name or None.

getLiftHintCmpr(valu, cmpr)

getLiftHintCmprCtor(name)

getStorCmprs(cmpr, valu)

getStorNode(form)

getTypeDef()

getTypeVals(valu)

isarray = False

merge(oldv, newv)

Allow types to “merge” data from two sources based on value precedence.

Parameters

• valu (object) – The current value.

• newv (object) – The updated value.

Returns

The merged value.

Return type

(object)

norm(valu)

Normalize the value for a given type.

Parameters

valu (obj) – The value to normalize.

Returns

The normalized valu, info tuple.

Return type

((obj,dict))

Notes

The info dictionary uses the following key conventions:

subs (dict): The normalized sub-fields as name: valu entries.

pack()

postTypeInit()

repr(norm)

Return a printable representation for the value. This may return a string or a tuple of values for display purposes.

setCmprCtor(name, func)

Set a comparator ctor for a given named comparison operation.

Parameters

• name (str) – Name of the comparison operation.

• func – Function which returns a comparator.

Notes

Comparator ctors should expect to get the right-hand-side of the comparison as their argument, and the returned function should expect to get the left hand side of the comparison and return a boolean from there.

setLiftHintCmprCtor(name, func)

setNormFunc(typo, func)

Register a normalizer function for a given python type.

Parameters

• typo (type) – A python type/class to normalize.

• func (function) – A callback which normalizes a python value.

stortype: int = None

synapse.lib.urlhelp module

synapse.lib.urlhelp.chopurl(url)

A sane “stand alone” url parser.

Example

info = chopurl(url)

synapse.lib.urlhelp.sanitizeUrl(url)

Returns a URL with the password (if present) replaced with ****

RFC 3986 3.2.1 ‘Applications should not render as clear text any data after the first colon (“:”) character found within a userinfo subcomponent unless the data after the colon is the empty string (indicating no password)’

Essentially, replace everything between the 2nd colon (if it exists) and the first succeeding at sign. Return the original string otherwise.

Note: this depends on this being a reasonably-well formatted URI that starts with a scheme (e.g. http) and ‘//:’ Failure of this condition yields the original string.

synapse.lib.version module

Synapse utilites for dealing with Semvar versioning. This includes the Synapse version information.

synapse.lib.version.fmtVersion(*vsnparts)

Join a string of parts together with a . separator.

Parameters

*vsnparts –

Returns:

synapse.lib.version.matches(vers, cmprvers)

Check if a version string matches a version comparison string.

synapse.lib.version.packVersion(major, minor=0, patch=0)

Pack a set of major/minor/patch integers into a single integer for storage.

Parameters

• major (int) – Major version level integer.

• minor (int) – Minor version level integer.

• patch (int) – Patch version level integer.

Returns

System normalized integer value to represent a software version.

Return type

int

synapse.lib.version.parseSemver(text)

Parse a Semantic Version string into is component parts.

Parameters

• text (str) – A text string to parse into semver components. This string has whitespace and leading ‘v’

• it. (characters stripped off of) –

Examples

Parse a string into it semvar parts:

parts = parseSemver('v1.2.3')

Returns

The dictionary will contain the keys ‘major’, ‘minor’ and ‘patch’ pointing to integer values. The dictionary may also contain keys for ‘build’ and ‘pre’ information if that data is parsed out of a semver string. None is returned if the string is not a valid Semver string.

Return type

dict

synapse.lib.version.parseVersionParts(text, seps=('.', '-', '_', '+'))

Extract a list of major/minor/version integer strings from a string.

Parameters

• text (str) – String to parse

• seps (tuple) – A tuple or list of separators to use when parsing the version string.

Examples

Parse a simple version string into a major and minor parts:

parts = parseVersionParts('1.2')

Parse a complex version string into a major and minor parts:

parts = parseVersionParts('wowsoft_1.2')

Parse a simple version string into a major, minor and patch parts. Parts after the “3.” are dropped from the results:

parts = parseVersionParts('1.2.3.4.5')

Notes

This attempts to brute force out integers from the version string by stripping any leading ascii letters and part separators, and then regexing out numeric parts optionally followed by part separators. It will stop at the first mixed-character part encountered. For example, “1.2-3a” would only parse out the “1” and “2” from the string.

Returns

Either a empty dictionary or dictionary containing up to three keys, ‘major’, ‘minor’ and ‘patch’.

Return type

dict

synapse.lib.version.reqVersion(valu, reqver, exc=<class 'synapse.exc.BadVersion'>, mesg='Provided version does not match required version.')

Require a given version tuple is valid for a given requirements string.

Parameters

• Optional[Tuple[int (valu) – Major, minor and patch value to check.

• int – Major, minor and patch value to check.

• int]] – Major, minor and patch value to check.

• reqver (str) – A requirements version string.

• exc (s_exc.SynErr) – The synerr class to raise.

• mesg (str) – The message to pass in the exception.

Returns

If the value is in bounds of minver and maxver.

Return type

None

Raises

s_exc.BadVersion – If a precondition is incorrect or a version value is out of bounds.

synapse.lib.version.unpackVersion(ver)

Unpack a system normalized integer representing a softare version into its component parts.

Parameters

ver (int) – System normalized integer value to unpack into a tuple.

Returns

A tuple containing the major, minor and patch values shifted out of the integer.

Return type

(int, int, int)

synapse.lib.view module

class synapse.lib.view.View

Bases: synapse.lib.nexus.Pusher

A view represents a cortex as seen from a specific set of layers.

The view class is used to implement Copy-On-Write layers as well as interact with a subset of the layers configured in a Cortex.

async addLayer(layriden, indx=None)

async addNodeEdits(edits, meta)

A telepath compatible way to apply node edits to a view.

NOTE: This does cause trigger execution.

async addTrigQueue(triginfo)

async addTrigger(tdef)

Adds a trigger to the view.

async callStorm(text, opts=None)

async callStormIface(name, todo)

async delTrigQueue(offs)

async delTrigger(iden)

async delete()

Delete the metadata for this view.

Note: this does not delete any layer storage.

async eval(text, opts=None)

Evaluate a storm query and yield Nodes only.

async finiTrigTask()

async fork(ldef=None, vdef=None)

Make a new view inheriting from this view with the same layers and a new write layer on top

Parameters

• ldef – layer parameter dict

• vdef – view parameter dict

• cortex.addLayer (Passed through to) –

Returns

new view object, with an iden the same as the new write layer iden

async getEdgeVerbs()

async getEdges(verb=None)

async getFormCounts()

async getStorNodes(buid)

Return a list of storage nodes for the given buid in layer order.

async getTrigger(iden)

init2()

We have a second round of initialization so the views can get a handle to their parents which might not be initialized yet

async initTrigTask()

isForkOf(viewiden)

isafork()

async iterStormPodes(text, opts=None)

async listTriggers()

List all the triggers in the view.

async merge(useriden=None, force=False)

Merge this view into its parent. All changes made to this view will be applied to the parent. Parent’s triggers will be run.

async mergeAllowed(user=None, force=False)

Check whether a user can merge a view into its parent.

async mergeStormIface(name, todo)

Allow an interface which specifies a generator use case to yield (priority, value) tuples and merge results from multiple generators yielded in ascending priority order.

async nodes(text, opts=None)

A simple non-streaming way to return a list of nodes.

async pack()

async runNodeAdd(node, view=None)

async runNodeDel(node, view=None)

async runPropSet(node, prop, oldv, view=None)

Handle when a prop set trigger event fired

async runTagAdd(node, tag, valu, view=None)

async runTagDel(node, tag, valu, view=None)

async scrapeIface(text, unique=False)

async setLayers(layers)

Set the view layers from a list of idens. NOTE: view layers are stored “top down” (the write layer is self.layers[0])

async setTriggerInfo(iden, name, valu)

async setViewInfo(name, valu)

Set a mutable view property.

async snap(user)

async classmethod snapctor(*args, **kwargs)

async storNodeEdits(edits, meta)

async storm(text, opts=None)

Evaluate a storm query and yield result messages. :Yields: ((str,dict)) – Storm messages.

async stormlist(text, opts=None)

class synapse.lib.view.ViewApi

Bases: synapse.lib.cell.CellApi

async getCellIden()

async getEditSize()

saveNodeEdits(edits, meta)

async storNodeEdits(edits, meta)

async syncNodeEdits2(offs, wait=True)