System Architecture¶

BDS makes use of data storage maintained through Amazon Web Services (AWS). Three AWS storage entities are required:

• A Kinesis data stream where BDS posts event objects.

• An S3 Bucket where BDS writes event objects if for some reason they can’t be written to the Kinesis stream.

• An SNS topic where BDS sends notifications. In particular, BDS sends a notification if BDS has to write events to the Bucket instead of the Kinesis stream.

All of these storage areas are maintained and paid for by the end user, not by D2L. This gives organizations the flexibility to configure and scale their storage spaces to match the size of their event stream.

Configuring BDS¶

The easiest way to configure BDS is to use the AWS command line interface. This involves downloading software tools and setting up security credentials as described in the AWS documentation.

When configuring BDS, configure the Bucket first, then configure the SNS topic, and finally configure the Kinesis stream.

BDS must be granted write permission on each of the areas. Users will be told how to grant this permission when they enroll for the BDS service.

BDS Event Posting¶

When a user interacts with Brightspace, Brightspace makes a record of what happened using a proprietary internal format. BDS takes data from Brightspace’s internal records and creates an event object in a standard format that can be processed by external users. This event object is posted to the end user’s Kinesis data stream.

Events are posted with minimal latency, so that event information may be obtained in close-to-real-time. Because the information undergoes some processing before events are posted, the event stream may not exactly match the chronological order in which the events actually occurred. However, each event is marked with a timestamp accurate to a thousandth of a second, so client-written software may reconstruct the temporal order of events if desired.

BDS only posts events that take place successfully. For example, if someone tries to perform an action but it fails because the user didn’t have appropriate authorization, the attempted action does not produce a BDS event.

Events are posted using JSON data objects as defined in RFC7159. At present, each such object conforms to the xAPI specifications. For more on the format of these event objects, see BDS xAPI event objects.

Brightspace Data Streams does not deliver every Brightspace event to BDS end users—some types of events are filtered out because they are unlikely to be of interest to organizations. For types of events that are passed on, BDS validates the event information before posting it to the Kinesis stream. BDS does not attempt to post an event unless it can be fully validated.

When BDS posts an event object to the Kinesis stream, the action activates an AWS Lambda function created by the end user. This function can process the event information in any way you wish. For example, it might copy the event object to some other location (either on your own computer network or another storage space in AWS). Later, you can mine the collected event objects for any data you like. You might also choose to analyze posted event objects so that you can take some immediate action in response; for example, if the teacher of a class posts a news announcement, your lambda could immediately notify the class’s students by text message or email.

By default, AWS automatically deletes event objects from a Kinesis stream 24 hours after the objects have been written. You can configure your stream so that data remains available for longer, up to seven days.

The Bucket¶

If for some reason an event object cannot be written to the Kinesis stream (e.g. if the stream is disabled, or if BDS doesn’t have permission to write events to the stream), BDS writes the event object to an AWS S3 bucket.

Event objects are held in the Bucket for seven days. During this time, BDS users can obtain objects from the Bucket and process them normally. Any events that remain in the Bucket after seven days will be automatically deleted.

Note that event objects in the Bucket aren’t invalid, they just couldn’t be delivered for some reason. For more on the Bucket, see The BDS Bucket.

Joining Event Object Data with Other Information¶

The xAPI event objects produced by BDS use UUIDs (Universally Unique Identifiers) to refer to people and objects. These UUIDs are produced by algorithms which always produce the same UUID for the same person or object.

BDS UUIDs make it easy for you to aggregate event data. For example, if you want to find all the events associated with a particular person, just search for events which contain that person’s UUID.

If you wish to determine specific information about the person or object associated with a UUID, you have to reconcile the UUID with identifying information stored by Brightspace. For example, if you want to know the name of a person associated with a UUID, you have to submit an appropriate query to Brightspace.

There are two general approaches for obtaining information from Brightspace:

• Individual lookups: The context part of every event object provides Brightspace-specific information. In particular, context specifies OrgUnit ID numbers, User ID numbers, and other identifiers related to the event. Using the Brightspace API, you can perform individual lookups on ID numbers and other identifiers to obtain associated information stored in Brightspace. For example, to obtain information about a particular user, you can use User API calls to look up information associated with a particular Brightspace ID number.

• Data set downloads: Simple API calls are useful for small-scale lookups, such as obtaining information on a single user. For large-scale lookups, such as obtaining information for all the students registered in a particular course, Brightspace’s Data export facilities are more appropriate. A data set contains information selected from multiple Brightspace records, potentially providing large quantities of data that meet specified criteria. Data sets are provided as Paged Data obtained through various API calls.

For more on processing event data, see Processing BDS Events.