Tyranid.js

Tyranid is a JavaScript-based, free, open-source, promise-based metadata framework for generically working with data.

Tyranid currently works primarily with MongoDB and can act as an ODM for MongoDB.

Status

The library is currently in its early stages and the API will likely change frequently until version 1.0 is reached.

Download

The repository is hosted at Github:

https://github.com/tyranid-org/tyranid

NPM:

npm install tyranid

Community

Tyranid has a Slack account for discussion and support for Tyranid. Click the button for an invite:

Guide

Setup

var mongodb = require('mongodb'),
    Tyr     = require('tyranid');

// Initialize Tyranid with a native MongoDB driver instance.
Tyr.config({
  db: await mongodb.MongoClient.connect(myDatabaseUrl)
});
... import/require all types and schemas:
// Once all the types and schemas have been processed, invoke Tyr.validate() to have Tyranid link and validate
// all the references between the types and schemas.
Tyr.validate();

// Optionally, you can pass in array of directory and filename regex match to Tyranid config to load the models and
// validate all at once.
Tyr.config({
  db: pmongo(myDatabaseUrl),
  validate: [{ dir: './app/models' }] // all files in directory
});
... or, with fileMatch:
Tyr.config({
  db: pmongo(myDatabaseUrl),
  validate: [{ dir: './app/models', fileMatch : '^tyr.*' }] // all files in directory that start with 'tyr'
});

Types

You can define your own types. Types can specify code to serialize to/from the browser, validation code, and so on.

new Tyr.Type({
  name: 'ssn',
  notes: 'Defines a U.S. Social Security Number',
  // SSNs will not be sent down to the browser by default for security reasons.
  // client can also be a function... 
  // --> client : function(value) { return this.security === 'low' } 
  // this will pass 'ssn' to the client if the value of 'security' in the containing document is 'low'
  client: false,
  validate: function(path, field, value) {
    if (!!value.match(/^\d{3}-\d{2}-\d{4}$/)) {
      return new Tyr.UserError({ field: path, suffix: 'is not a valid SSN.' });
    }
  }
});

Schemas

// Metadata describing nested structures can be either placed inline or defined separately.
var RoleEntry = {
  is: 'object',
  fields: {
    network: { link: 'network' },
    role:    { link: 'role' }
  }
};

var User = new Tyr.Collection({
  // Collections also have an id which is used to construct universal ids and other metadata purposes.
  id: 'u00',
  name: 'user',
  dbName: 'users', // defaults to value for "name"
  fields: {
    // Types like "email", "mongoid", "password", etc. are built in to Tyranid but can be overridden.
    _id:        { is: 'mongoid' },
    userId:     { is: 'mongoid', label: 'User ID' },
    email:      { is: 'email' },
    // Shorthand:  "'password'" is equivalent to "{ is: 'password' }"
    password:   'password',
    // required: true will make this field required during validation.
    firstName:  { is: 'string', required: true },
    lastName:   { is: 'string' },
    canLogIn:   { is: 'boolean' },
    birthDate:  { is: 'date' },
    ssn:        { is: 'ssn' },
    // "org" links to another collection "organization".  is does not need to be specified -- Tyranid will infer
    // it based on the data type of the _id field in "organization".  Tyranid will also validate the schema and ensure that
    // it knows what "organization" is and update the reference with the Organization instance.
    org:        { link: 'organization',
                  // as can be applied to fields or collections to give a human-readable name suitable for using in a UI.
                  label: 'Organization'
                },
    inactive:   { is: 'boolean' },
    comments:   { is: 'string',
                  // help contains user-facing documentation that could be used in, for example, a tool tip.
                  help: 'Place any public comments you would like to make here.',
                  // note contains internal, developer documentation.
                  notes: 'Used for various notes that the user sets themselves.' },
    roles:      { is: 'array', of: _.cloneDeep(RoleEntry) }
  },
  // Can optionally specify a field to act as this collection's primary key, defaulting to _id.
  // This field will be used for document retrieval, validation, and, most importantly, links and population.
  // Setting defaultMatchIdOnInsert will cause Tyranid to match _id to this field's value on
  // insert (instead of generating a new ObjectId).
  primaryKey: { field: 'userId', defaultMatchIdOnInsert: true }
});

User.myStaticMethod = function() { ... };

User.prototype.fullName = function {
  return this.firstName + ' ' + this.lastName;
};

// Tyranid can fetch collections by name.
(Tyr.byName['user'] instanceof Tyr.Collection) === true

Mixin Schemas

Tyranid also supports mixin schemas via the Collection.mixin() method that can statically add metadata to an existing collection. This is useful for extending built-in Tyranid collection with additional metadata to suit your needs.

The following adds a "westTotal" property to User:

User.mixin({
  def: {
    fields: {
      westTotal: { is: 'integer' }
    }
  }
});

Dynamic Schemas

Tyranid also supports dynamic schemas that can be added at run time via storing metadata in the database. Dynamic schemas differ from Mixin Schemas in that they are conditionally applied based on the match rules.

The following adds a "westTotal" property to User but only for users that belong to Organization 1:

Tyr.byName.tyrSchema.save({
  collection: User.id,
  match: {
    org: 1
  },
  type: Tyr.byName.tyrSchemaType.PARTIAL._id,
  def: {
    fields: {
      westTotal: { is: 'integer' }
    }
  }
});

You can get a list of fields that apply to a specific document using fieldsFor:

const fields = await User.fieldsFor({ match: { org: 1 } });

Finding & Saving

var user = new User();
user.firstName = 'David';

// Built-in document methods and properties start with $ to avoid collisions with actual document property names.
// (It is illegal to have properties that start with $ in MongoDB.)
user.$save().then(function() {
  // save is complete
});

// Document level update, but only saves the shallow properties of the object, if the properties are set in the object.
user.$update().then(function() {
  // update is complete
});

// Collection level update, same API as mongo shell update().
User.update(query, update, opts).then(function() {
  // update is complete
});

// collection.db is an instance of a native MongoDB collection for when it is easier to just work directly with mongo.
User.db.find({ ... }).then(function(users) {
});

// Similar to the previous line except that users are wrapped as User objects.
User.find({ ... }).then(function(users) {
});

// Returns a single user wrapped as a User object.  Parameters are the same as MongoDB's findOne() API.
User.findOne({ ... }).then(function(users) {
});

// Behaves the same as MongoDB's findAndModify() API.  If using timestamps and the update option, Tyranid will add in
// an updatedAt update as well.
User.findAndModify({ ... }).then(function(users) {
});

// Returns the user with ObjectId('AF8...123') wrapped as a User object.  byId(id) is a shortcut for
// User.findOne({ _id: id }).
User.byId(ObjectId('AF8...123')).then(function(user) {
});

// Same as User.db.remove()
User.remove({ _id: id });

Counting

Tyranid supports counting operations to make implementing things like paging easier.

const childCount = await User.count({ query: { age: { $lt: 18 } } });

...

const page = Document.findAll({
  query: { myAttribute: false },
  skip: 30,    // page 4
  limit: 10,   // 10 rows per page
  count: true
});

page.length // 10 (or less if on the last page)
page.count  // n where n equals the total number of documents in the collection with myAttribute === false

Labels

Labels can be defined for documents using the labelField option:

var Continent = new Tyr.Collection({
  ...
  fields: {
    ...
    // labelField indicates that this field is the label value for this collection.
    name:       { is: 'string', labelField: true },
    ...
  }
});

Examples using labels are:

// Finds the document in an emum collection where the label is equal to the given label.
let country = Country.byLabel('Germany');

// Returns the value of the label field for the document.  The following would output "Germany":
Country.labelFor(country)

// The same method is available on document instances as a dynamic property as well:
country.$label

Collections and Fields also support a label property. If no label is provided one will be generated from the name.

User.label                    => 'User'
User.def.fields.userId.label  => 'User ID'
User.def.fields.email.label   => 'Email'

Fields

Fields support a lot of properties -- see Field for a complete list.

// Fields support a name property.
User.def.fields.email.name => 'email'

// Fields also support a path property.
User.def.fields.roles.def.of.def.fields.network.path => 'roles.network'

Conditional Fields

All Fields also support an if property which can help model things like unions. For example:

const Animal = new Tyr.Collection({
  name: 'animal',
  ...,
  fields: {
    ...
    hasLegs: { is: 'boolean' },
    // The legCount property only exists if hasLegs is true.
    legCount: { is: 'integer', if: { hasLegs: true } },
    ...
  }
});

A field's if property can also be an (optionally asynchronous) function that returns either a matching criteria or a boolean. For example:

var User = new Tyr.Collection({
  id : 'u00',
  name: 'user',
  fields: {
    ...
    type: { link: 'animalType' },
    legCount: { is: 'integer',
                // this function can return a promise
                if(opts) { return determineIfHasLegTypes(this) }
              }
    ...
  }
});

The opts parameter is populated with relevant contextual information that is available:

Value	Notes
auth	The currently logged-in user making the request. Also available as user.
req	The express Request if the call originated from the client.
user	The currently logged-in user making the request. Also available as auth.

This feature is analogous to link constraints.

Field Groups

Tyranid supports a concept called field groups that allow you to reduce repetitive schema definitions.

The following:

const Widget = new Tyr.Collection({
  ...
  fields: {
    ...,
    super: { is: 'boolean' },
    $super: {
      $base: { if: { super: true }, note: 'Only super widgets have this field.' },

      superName: { is: 'string' },
      superAccess: { is: 'integer' }
    }
  },
});

is equivalent to:

const Widget = new Tyr.Collection({
  ...
  fields: {
    ...,
    super: { is: 'boolean' },
    superName: { is: 'string', group: '$super', if: { super: true }, note: 'Only super widgets have this field.' },
    superAccess: { is: 'integer', group: '$super', if: { super: true }, note: 'Only super widgets have this field.' }
  },
});

All properties present in the $base property are copied to all of the fields in the group and a group property is added to each field which contains the name of the group.

Links

Links are fields that connect collections together. Links are also known as foreign keys.

Link Relationships

Fields of type link support a relate property. The valid values for relate are:

Value	Notes
associate	The default. Indicates a regular associative link.
owns	Indicates that the linked-to document is "owned" or "composed in" the linking document.
ownedBy	Indicates that the linked-to document "owns" or "is composed of" the linking document.

For example:

var Department = new Tyr.Collection({
  id: 'd00',
  name: 'department',
  fields: {
    ...
    // This ties departments to the lifecycle of their organization.
    organization:   { link: 'organization', relate: 'ownedBy' },
    ...
  }
});

Link Constraints

link Fields also support a where property. For example:

var User = new Tyr.Collection({
  id : 'u00',
  name: 'user',
  fields: {
    ...
    // Managers have to be in the same organization as the user.
    manager:   { link: 'user', where: { organizationId: $this.organizationId } },
    ...
  }
});

The link's where property can also be an (optionally asynchronous) function. For example:

var User = new Tyr.Collection({
  id : 'u00',
  name: 'user',
  fields: {
    ...
    // Managers have to be in the same organization as the user.
    manager:   { link: 'user',
                 async where(opts) {
                   return {
                     organizationId: { $in: await determineAvailableManagersByRole(this.role) }
                   };
                 }
               }
    ...
  }
});

The opts parameter is populated with relevant contextual information that is available:

Value	Notes
auth	The currently logged-in user making the request. Also available as user.
req	The express Request if the call originated from the client.
user	The currently logged-in user making the request. Also available as auth.

This feature is analogous to conditional fields.

Optional Links

link Fields can also be optional. In the following example, if the project collection does not exist then the link project will be pruned from the schema for User at run time:

var User = new Tyr.Collection({
  id : 'u00',
  name: 'user',
  fields: {
    ...
    project: { link: 'project?' }
    ...
  }
});

Optional links are useful for when you want to define generic code that gets additional capabilities if certain collections are present. For example, Tyranid's logging system has an optional link to "user". If a user collection is present, it will be used, but if not, then the user link will be pruned from the log schema at run time.

Link Methods

Collections support a Collection.links() method to search for incoming and outgoing links.

Client

Tyranid will automatically generate collections, fields, and so on for use on the client based upon the server metadata.

Only collections that have client: true will have their metadata sent to the client. By default collections are server-only.

Client-side Tyranid is enabled by the following two steps:

1. Initialize routing. express is the instance returned by express(). auth is a callback that will be placed ahead of all routes (you can do additional authorization in this function).

   const expressApp = require('express')(),
         session = require('express-session'),
         MongoStore = require('connect-mongo')(session),
         store = new MongoStore({ url: myUrl }),
         httpServer = expressApp.listen(80);
   
   ...
   
   Tyr.connect({
     app: expressApp,
     auth: (req, res, next) => { ... },
     http: httpServer,
     store: store
   });

2. Import client-side code into your browser session:

   <script src="/api/tyranid"></script>

The Tyranid namespace is available on the client as window.Tyr.

Methods, properties, and classes that are available on the client are identified in the documentation by the client tag:

Code will be transpiled to ES5 by Babel, so you can use ES2015 functionality in your client-side code.

Code generated will also by default be minified (controlled by the minify option in Tyr.config()).

Computed Properties

Computed values can be created by specifying a get option on a property. By default computed values will:

1. not be stored in the database unless the db property is set to true, and
2. not be sent down to the client with toClient() unless the client property is set to true.

var User = new Tyr.Collection({
  ...
  fields: {
    ...
    firstName:  { is: 'string' },
    lastName:   { is: 'string' },
    fullName:   { is: 'string', get: function() { return this.firstName + ' ' + this.lastName; }, db: true, client: true },
    ...
  }
});

By default get methods are isomorphic if client-side functionality is enabled. If you want the function to only appear on the server you can use getServer instead. You can also optionally specify a client-specific version using getClient.

Set

set, setClient, setServer also exist to provide a programmatic setter for properties as well.

Methods

Tyranid supports defining methods in metadata.

var User = new Tyr.Collection({
  ...
  methods: {
    methodName: {
      params: {
        param1: param field definition,
        ...,
        paramN: param field definition
      },
      return: return field definition,
      fn(parameters) { ... },
    },
    ...
  },
  ...
});

By default fn methods are isomorphic if client-side functionality is enabled. If you want the method to only appear on the server you can use fnServer instead. You can also optionally specify a client-specific version using fnClient.

Methods allow you to run the same code on the server and the client. If what you want to do is run the code on the server but call the code from the client look at Services.

Projection

All fields options on options support an extended syntax besides standard MongoDB projections.

If fields is given a object, then it should be a standard MongoDB-style projection.

If fields is given a string, then this string should refer to the name of a predefined projection defined on a collection's projections property. For example:

var User = new Tyr.Collection({
  ...
  projections: {
    ...
    myProjection: {
      name: 1,
      age: 1
    },
    ...
  },
  ...
});

// The following projection is equivalent to:  { fields: { name: 1, age: 1 } }
const u = User.byId(myId, { fields: 'myProjection' });

If fields is given an array, then the array should contain a list of projections that should be merged. For example:

// The following projection is equivalent to:  { fields: { name: 1, age: 1, ssn: 1 } }
const u = User.byId(myId, { fields: ['myProjection', { ssn: 1 }] });

Note that _id is included by default in all projections unless _id: 0 appears.

Client Views

Pre-defined projections can also be used to define common client-side views.

var User = new Tyr.Collection({
  ...
  projections: {
    mainTableView: {
      name: 1,
      job: 1,
      organization: 1
    },
    ...
  },
  ...
});
...
const myUser = await User.byId(myId, { fields: 'mainTableView' });
res.json(myUser.$toClient({ fields: 'mainTableView' }));

Minimal Projections

There is a special named collection called $minimal. If this named projection is present then these fields will always be included in the projection even if they are not directly requested. You can override this behavior by passing in $minimal: falsey value into your projection.

The $minimal projection is applied first, then the manually specified projection is merged on top of it, so any fields mentioned in the manual projection override the corresponding fields in the $minimal projection.

For example, given:

var User = new Tyr.Collection({
  ...
  projections: {
    ...
    $minimal: {
      name: 1,
      age: 1
    },
    ...
  },
  ...
});

The following query:

User.find({ query: { job: 1 } })

would have an effective projection of { name: 1, age: 1, job: 1 }, while the following query:

User.find({ query: { job: 1, $minimal: false } })

would have an effective projection of just { job: 1 }.

Population

// Tyranid can traverse links and provide join-like functionality.
User.populate('org', users).then(function(users) {
  // An org$ property on each user will be added with an actual organization document.
});

// If your field that you are populating ends with "Id", the populated field will be the same name with the "Id" removed
// instead of a "$" appended.
User.populate('orgId', users).then(function(users) {
  // An org property on each user will be added with an actual organization document.
});

// If you leave off the second array of documents parameter to populate() it will return a curried version of itself
// that can be inserted into a find promise chain.
User.find({ ... }).then(User.populate('org'));

// You can also populate a single instance:
user.$populate('org');

// You can also pass in pathnames to populate.  Arrays will also be populated as well.
user.$populate('roles.role');

// You can also pass in an array of property names to populate.
user.$populate([ 'org', 'roles.role' ]);

Advanced Population

The advanced form of population allows you to perform nested populations and specify the projections of populations.

// Population often uses an "$all" constant.  "Tyr.$all" is defined as "Tyr.$all = '$all'".
var Tyr  = require('tyranid'),
    $all = Tyr.$all;

// The following says to populate the org field and also populate the "_id", "name", and "network" properties from role.
// Additionally, the network$ property on roles will also be populated, but only the names will be retrieved, not the _id's
// (or any other properties ... _id's are included by default in projections according to MongoDB conventions).
user.$populate({ org: $all, 'roles.role': { name: 1, network: { _id: 0, name: 1 } } });

// The following says to grab all the properties off of role and to also populate the network property.
user.$populate({ 'roles.role': { $all: 1, network: $all } });

// "$all: 1" can also be written "$all: $all".  If using ES6, this means it can be written as just "$all".
// For example the previous can be written as:
user.$populate({ 'roles.role': { $all, network: $all } });

The values that can follow a property pathname are:

0	Means exclude this property. Similar to MongoDB's projection.
1	Means include this property. Similar to MongoDB's projection.
$all	Means to populate the field and grab all the properties. Can only be specified on a link.
{ ... }	Recursively list properties to populate. Can only be specified on a link.

In general, population will not query the same document more than once. If the document shows up in multiple places with different projections all locations will contain the same document which will contain a superset of the projections. This also means that population can be used to populate relationships with cycles.

Advanced population syntax also supports the predefined projection and projection merging syntax. For example:

// This populates the owner using the user collection's predefined 'nameAndTable' view.
job.$populate({ owner: 'nameAndTable' });

// Another example using projection merging:
job.$populate({ owner: ['nameAndTable', { ssn: 1 }] });

Denormalization

Tyranid can also store populations when a document is saved or updated. For example, this can be useful for cases where you need a a few properties redundantly stored in a collection so you have more indexing options.

This capability shows up as a denormal option when defining a field. For example, the following would store a copy of the Organization's name field redundantly on the User collection. Similar to how population would store populated org data under a variable "org$", denormalized data will get stored as "org_". In the following example, the org link would be available as "user.org" while the denormalized name would be available as "user.org_.name".

var User = new Tyr.Collection({
  ...
  fields: {
    ...
    org:  { link: 'org', denormal: { name: 1 } },
    ...
  }
});

Validation

The following will run validations on the object and return an array of UserErrors:

user.$validate();

Timestamps

If you set timestamps: true when defining your collection then Tyranid will update the timestamps createdAt and updatedAt when you create a new object using Tyranid's save() or insert() methods and it will update updatedAt when you use Tyranid's update() methods.

var MyCollection = new Tyr.Collection({
  ...
  timestamps: true,
  ...
});

Most Tyranid saving and update methods also support an option to disable updating timestamps for the specific call. For example:

myDoc.$update({ timestamps: false });

Maps

You can define your objects to be maps using the keys and of field properties.

For example, the following defines a map of strings to integers.

{
  is: 'object',
  keys: 'string',
  of: 'integer'
}

As another more complex example, the following defines a map of UIDs to objects.

{
  is: 'object',
  keys: {
    is: 'uid',
    of: [ 'user', 'group' ]
  },
  of: {
    is: 'array',
    of: 'string'
  }
}

You can also combine fields and keys/of -- for example, the following defines a simple named array-like object of strings:

{
  is: 'object',
  fields: {
    name:   { is: 'string', fieldLabel: true },
    length: { is: 'integer' }
  },
  keys: 'number',
  of: 'string'
}

Universal IDs (UIDs)

A UID is a string formed by concatenating a collection ID with a document ID.

Universal IDs are a way of polymorphically working with data. For example, if you need to have an array of IDs from multiple collections UIDs will let you do this.

// Returns "u00AF8...123".  u00 was specified as the collection ID for User.
User.idToUid(ObjectId('AF8...123'))

// returns "u00AF8...123" if user._id is ObjectId("AF8...123").
user.$uid

// returns { collection: User, id: ObjectId('AF8...123') }
Tyr.parseUid('u00AF8...123')

// Tyranid can fetch documents by UID without having to specify the collection for code that generically works with data.
Tyr.byUid('u00AF8...123').then(function(user) {
  // user is a User instance with mongo ID ObjectId('AF8...123').
});

// Tyranid also efficiently fetch an array of documents by UIDs as well:
Tyr.byUids(['u00AF8...123', 'g00FDC...321', ...]).then(function(docs) {
  ...
});

Browser Serialization

// Returns a User instance while also performing security checks, converting the _id to a MongoId, validating data,
// and so on.
User.fromClient({ _id: 'AF8...123', firstName: 'Jack', ... })

// Returns a "Plain Old JavaScript Object" (POJO)/JSON object that is suitable for sending down to client while also
// performing security checks.
res.json(user.$toClient())

// Translates an existing object to the client when the object is not an instance of User (i.e. came directly from Mongo).
res.json(User.toClient(obj))

Paths

Paths are a string-based syntax for referencing fields inside documents. For example, the following structure:

{
  name: 'Jack',
  siblings: [
    { name: 'Jill' },
    { name: 'Joe' }
  ],
  address: {
    state: 'WI'
  }
}

has the following paths:

name	'Jack'
siblings.0.name	'Jill'
siblings._.name	'Jill', 'Joe'
address.state	'WI'

Collections contain a paths hash which make it easier to work with paths.

Arrays and Maps

Arrays and maps have a special syntax. When referring to the contents of an array or map in a generic sense, you use "._". When referring to a particular element you use ".array index | map key".

For example, siblings refers to an array, siblings._ refers to the type of contents of the array (an object), and siblings._.name refers to a specific property inside that object that is inside the array.

This array syntax can be simplified (i.e. siblings._.name => siblings.name) in certain circumstances, for example when used inside MongoDB queries. This is called a simplified path and is available as Field.spath.

Path

The collection.parsePath(path) method can be used to create Path objects which are useful for parsing paths.

Path Labels

Fields have a pathLabel property which can be used to specify an abbreviated label a field when it is used as part of a full-path label. For example, if your path was name.first, you could set the pathLabel on name to be blank and set the pathLabel to "First Name" on name.first. This would cause name.first to have a full path label of "First Name" instead of "Name First". This property can also be used to set abbreviated names for path nodes so that the overall path label is not too long.

Singleton Collections

Singleton collections also have a special path syntax -- collection name:path name.

Subscriptions

If socket.io functionality has been enabled by connect()ing an express http server to Tyranid, then client subscription functionality is enabled.

Each client can issue a Collection.subscribe() call to express interest in a query of data that they would like to be kept live. As this data is updated by Tyranid calls on servers, any documents matching the subscribed query will be pushed down to the client via WebSocket connections.

Clients can listen to updates on a collection by using Collection.on().

The data that Tyranid is currently keeping live is kept at Collection.values.

Often times you will have server calls that bypass Tyranid client methods that will return data to the client. You can update Tyranid's local copy of data with this new information using the Collection.cache() method.

Validating Subscriptions

When a subscription is requested a subscribe event is sent to the collection being subscribed to.

The event's query field has the query of the subscription and the event's opts.auth property has the user making the request. The query and user can be examined and then a SecureError raised if the query is not allowed.

The event also has a subscription field which contains the pending subscription record. Rather than raising an exception you can also just modify this record in place (though note that the query needs to be JSON.stringify()'ed).

For example:

MyCollection.on({
  type: 'subscribe',
  handler: event => {
    console.log('The user making the subscription is: ', event.opts.auth);
    console.log('The query the user is subscribing to is: ', event.query);
    console.log('The pending subscription is at: ', event.subscription);
    if (... fails validation ...) {
      throw new Tyr.SecureError('This subscription is not allowed.');
    }

    // mutate query example:
    event.query.user = event.opts.auth._id; // user can only subscribe to stuff they own
    event.subscription.query = JSON.stringify(event.query);
  }
});

Note that regular security checks also apply to subscriptions, so subscription validation as shown here is not always necessary.

Historical Data

Tyranid's Historical Data support modifies the built-in methods like $save() and so on to keep track of changes to documents so that historical values are kept.

Historical data is enabled by specifying the historical option on collections.

Once it is enabled on a collection, you must also mark which fields should historical using the historical option on fields.

Note that only top-level fields can be marked historical -- anything contained beneath a top-level field is automatically assumed to also be historical.

Formats

There are two formats that historical data can be stored in -- document and patch. The two formats have the following tradeoffs:

Type	Storage of History	Document Size	Database Size	Searchable via Database queries
document	copies of the document as it appeared at different times stored in the separate collectionDatabaseName_history collection	Small	Large	Yes*
patch	array of patches stored on the document itself in its _history property	Large	Small	No

* The queries must take into account _partial snapshots.

Document Format

The document format stores history in a separate collection named collectionDatabaseName_history.

The format of the collectionDatabaseName_history collection is identical to the format of the underlying collection with the two differences:

• only fields that have the historical property set to true are stored in the snapshot
• the following fields are added:

  Name	Contains	Optional
  __id	the _id of document this is a snapshot of	No
  _on	the date of this snapshot in history	No
  _author	the ID of the author of this change	Yes
  _comment	notes made about this change	Yes
  _partial	whether this is a partial update (i.e. does not contain the full document)	Yes

Patch Format

The patch format stores history on the document itself in a _history property which contains an array of the changes.

Example

const update = User.fromClient(req.body),
        orig = await User.byId(update._id); // historical database reads will update the document's $orig property

orig.$copy(update); // $copy() will not overwrite orig's $orig (and _history if patching) properties
await orig.$save(); // $save() will diff orig.$orig and orig and either
                    //   update orig._history with a new snapshot entry (patch) or
                    //   insert a record into the collectionDatabaseName_history (document)

Also see $replace() as an alternative to $copy().

If you want to update just part of a document and do not want to read in the whole thing, you can do:

doc = await User.byId(myId, { fields: { age: 1 } });
doc.age = 38;
await doc.$update(); // the _history field will still be efficiently updated even though it is not present in the query

Or, if you have a more complete document but only want to update part of it:

doc = await User.findOne({ query: { ssn: mySsn } });
doc.age = 38;
await doc.$update({ fields: { age: 1 } });

All historical-compatible methods also support specifying the author and comments:

doc = await User.findOne({ query: { ssn: mySsn } });
doc.age = 38;
await doc.$update({ fields: { age: 37 }, author: req.currentUser, comment: 'The age had a typo' });

Most query methods support an asOf option that also works with population so that a hierarchy of historical documents can be retrieved as of a particular date. In the following example, populated department records are also as of the given date:

doc = await User.findAll({
  query: { ssn: mySsn },
  asOf: new Date('2016-05-03'),
  populate: { department: $all }
});

Patch-specific Methods

If you just want to push or pull data onto an array and maintain historical information, see Collection.push() and Collection.pull().

$snapshot() is also available to perform the differencing and create a snapshot in memory. This can be useful if you want to make a number of snapshots in memory and write them out at once (bulk updates for many documents, single updates when importing history from another source, and so on).

You can revert a document to older versions using the $asOf() method:

doc = await User.byId(myId);

doc.$asOf(new Date('2016-05-03')); // revert doc to what it looked like on May 3, 2016.

Documents reverted using $asOf() become read-only, as updating a document in the past does not make sense and these documents will have a non-enumerable $historical property added to them to indicate this state.

Warning

If you bypass Tyranid calls and use the native MongoDB driver calls or modify database directly in the database, historical values will not be updated.

The current historical-safe manipulation methods are:

• Collection.insert()
• Collection.pull()
• Collection.push()
• Collection.save()
• Collection.updateDoc()
• Document.$save()
• Document.$update()

Historical-unsafe methods like Collection.findAndModify() will log a warning message if they are used with historical collections. You can suppress this warning by passing in "historical: false" in the options object.

Other Approaches

Tyranid's built-in support for historical data is by no means the only -- or necessarily the best -- way to deal with historical data. As one example, if you can structure your data to be immutable then this mechanism is unnecessary.

Static Data

Tyranid supports efficiently modeling static data as well. This is good for modeling enumerations and linking to them. It also allows you to work with static and non-static data in an isomorphic way.

var Continent = new Tyr.Collection({
  id: 'c0a',
  name: 'continent',
  dbName: 'continents',
  // This causes enumeration constants to be defined on the collection for each of the enumerated values.
  enum: true,
  fields: {
    _id:        { is: 'integer' },
    // labelField indicates that this field is the label value for this collection.
    name:       { is: 'string', labelField: true },
    code:       { is: 'string' },
    altName:    { is: 'string', label: 'Alternate Name' }
  },
  values: [
    [ '_id', 'name',       'code', 'altName' ],

    [ 4328, 'Africa',        'AF', null      ],
    [ 4329, 'Antartica',     'AN', null      ],
    [ 4330, 'Asia',          'AS', null      ],
    [ 4331, 'Europe',        'EU', null      ],
    [ 4332, 'North America', 'NA', 'NA'      ],
    [ 4333, 'Oceania',       'OC', null      ],
    [ 4334, 'South America', 'SA', 'SA'      ]
  ]
});

Continent.prototype.isContinent = function() {
  return true;
};

// This evaluates to true since the enum flag was turned on.  "NORTH_AMERICA" is derived from the name "North America"
// since the name field was marked as the labelField for the Continent collection.
Continent.NORTH_AMERICA._id === 4332

// The static data instances are also instances of the collection so methods and behaviors are available.
Continent.NORTH_AMERICA.isContinent()

If your static data has fields which link to another static collection, you can use labels in the values as the foreign key to make the data more readable (they will be converted to ids when the collection is validated).

If you have static data that does not have a unique label field, you can mark your data as being static instead of enum.

Units

Units of Measurement are an important piece of metadata that Tyranid tracks. In addition, Tyranid has a lot of knowledge around how to manipulate and do arithmetic with units.

You can associate units with a field using the in option:

var Widget = new Tyr.Collection({
  id: 'w00',
  name: 'widget',
  ...
  fields: {
    ...
    // m^3 means m3 or meters cubed and indicates a volume tracked in meters.  Can also be written as "m3".
    volume: { is: 'double', in: 'm^3' }
    ...
  },
  ...
});

The Units for fields are accessible on the Field objects:

var units = Widget.fields.volume.in;

You can also create your own Units instances using Units.parse() or Tyr.U.

const U = Tyr.U;

kg = U('kg');

// The parsing functions can also be used as tagged template strings:
kg = U`kg`;

// Tyranid also supports composite units:
mph = U`mi/h`;

// Units can infer information about types and systems:
U`m/s`.type.name === 'velocity'
U`m/s`.system.name === 'metric'
U`m/s2`.type.name === 'acceleration'
U`m*kg/s2`.type.name === 'force'
U`mP`.system.name === 'planck'
U`ft`.system.name === 'english'

// Units can do conversions:
U`in`.convert(12, U`ft`) === 1

// Units can do addition and subtraction:
U`in`.add(5, U`ft`, 1) === 17
U`in`.subtract(12, U`ft`, 1) === 0

// Units can do multiplication, division, and inversions:
U`m`.multiply(U`m*s-1`) === U`m2/s`
U`m`.divide(U`s2`) === U`m/s2`
U`m/s`.invert() === U`s/m`

Currency Conversion

Tyranid optionally uses Fixer to look up current exchange rates to perform currency conversion. To enable this create a Fixer account and pass in your API key to Tyr.config().

See Units and Unit for more information.

License

The project is licensed under Apache License, Version 2.0.

Patches are gladly accepted from their original author. Along with any patches, please state that the patch is your original work and that you license the work to the Tyranid project under the Apache License, Version 2.0 open-source license.

Further Information

If you have any questions, please send email to info@tyranid.org.