Alexa

Automatic monitoring of AWS Lambda functions for an Alexa Skill

If you’re not testing your services in production yourself, then you’re letting your customers test it for you!

Automatic testing of your system means you can catch issues before your customers see them. This doesn’t mean you can skip on unit testing or functional testing, it simply means you have an extra layer to notify you of problems before they affect users.

What to test

The Alexa Skill that I want to test is implemented as an AWS Lambda Function. The Lambda Function will be the focus of the automated testing since that’s where my code lives and where it’s most likely to have a problem. I’m not testing anything to do with voice commands, Echo devices, or the Alexa Skill backend. I’m treating that as a black box, after all I don’t have any way of fixing or debugging issues inside of those systems anyways.

When I’m monitoring the system I want to know the following

  • Is the Lambda function online and working
  • Are dependencies functioning correctly
  • How long does it take to execute (latency)

To test this I’ve decided to take the simplistic approach and use an automated request to my Lambda Function that synthesises a customer request and then use my monitoring systems to identify problems.

This works well for my use case because I have already implemented monitoring and logging for Lambda Function so I have pretty good metrics and most users interact with my skill on a weekly basis so this is likely to highlight problems before users find them.

How to test it

My Lambda function exposes a single handler function in Node.js so to test different behaviours you have to modify the values sent in the request payload (rather than having different APIs for each). There are a couple of different options that I can use to test my function but for simplicity and cost it’s easier to use an AWS CloudWatch Event to automatically trigger my Lambda Function at a set interval and check everything is OK.

Setting up the Cloudwatch event is split into two parts:

  1. Modifying the Lambda Function to allow CloudWatch
  2. Setting up the Event itself to call the function at a set interval

Allowing the function to be invoked is pretty simple. You modify your Lambda Function and add a new trigger from the menu on the left for “CloudWatch Events”, This means your function can be invoked either from an Alexa Skill or from CloudWatch.

You can then go into CloudWatch and add a new event. I’ve opted to trigger this event on a set schedule every 5 minutes. You can also select the target as being a Lambda function and then select the function name from the list, and also the version of the function you want to target (I always select the same version that I have live for users to make sure I’m testing what users are experiencing).

For the input of the event, I used a fixed JSON payload which specifies the intent name that I want to trigger.

{   
  "session": {     
    "new": true,     
    "sessionId": "SessionId.253d1fe4-9af0-45de-b767-ccfea6f0e3d4",     
    "application": {       
      "applicationId": "<YOUR SKILL ID HERE>"     
    },     
    "attributes": {},     
    "user": {      
      "userId": "HEALTH-CHECK-USER"     
    }   
  },   
  "request": {     
    "type": "IntentRequest",     
    "requestId": "EdwRequestId.cb85ea9c-1e57-4226-83b6-f1a1d9e2eb8a",     
    "intent": {      
      "name": "PlayLatestSermon",       
      "slots": {}     
    },     
    "locale": "en-GB",     
    "timestamp": "2018-01-14T14:31:58Z"   
  },   
  "context": {     
    "AudioPlayer": {       
      "playerActivity": "IDLE"     
    },     
    "System": {       
      "application": {         
        "applicationId": "<YOUR SKILL ID HERE>"       
      },        
      "user": {         
        "userId": "HEALTH-CHECK-USER"       
      },       
      "device": {         
        "supportedInterfaces": {}       
      }     
    }   
  },   
  "version": "1.0" 
}

Once that’s in place you should have the specified intent being triggered every 5 mins!

Knowing when things go wrong

I’ve already written about monitoring Alexa Skills and creating a dashboard. I’d rather not have to keep checking graphs to know when something is going wrong. So, you can actually use CloudWatch alarms to check metrics for you and send an email when a configured threshold is breached.

When you add a CloudWatch event it automatically logs some metrics for each configured event, including for successful and failed invocations. This is perfect for knowing when the health check failed. I followed the configuration options for the alarm and set this to fail if I have three or more failed invocations in a 15 min period. I’ve also configured some other alarms on errors, and some capacity alarms. One thing that I find helpful is to set an alert when the status is ALARM (when it goes wrong) and also when it’s OK (when it recovers), that way, if you get a blip that triggers the alarm, you’ll also get a follow up telling you it was OK.

The beauty of this approach is that you get automatic traffic testing your code at whatever interval you pick, and you also get notified when something starts to misbehave so you can catch it before your users do helping ensure you have a reliable system and a better experience for your users

Using AWS CloudWatch to monitor an Alexa Skill (Node.js)

Monitoring Alexa Skills can be a bit hit or miss. On one hand, you get some metrics for free (throttles, invocations, duration, errors) which is great, but sending custom metrics can be a bit confusing, and it’s not helped by the 166 npm packages that you find when you search for ‘cloudwatch’ in the npm database.

What I want is a way of monitoring different parts of my Alexa skill, especially:

  • Execution time
  • RSS feed download time
  • RSS feed parsing time
  • Custom errors
  • Which intents are being used the most

Sending custom metrics to CloudWatch via Metric Filters

After looking through the various NPM packages, and the CloudWatch API. Thankfully I found a much easier way to push custom metrics to CloudWatch. These are called Metric Filters. They allow you to set up parsing rules for CloudWatch log lines, when a log line matches the filter it can be pushed as a metric as well (and then graphed). This means you can graph metrics, just by writing a log line in a consistent format!

Log Format

The log format I settled on is pretty simple, I have two main metric types:

  • Timers – These log a value (duration) against a specific metric name
  • Counters – These just log out a metric name and can be used to count the number of times something happened (perfect for errors).

I wrapped this format in a simple logger object so I can just call logger.error(), logger.timer() or logger.increment() in my code and pass the name (and value if it’s a timer).

var logger = {

    timer: function(metricName, value) {
        console.log("TIMER_METRIC " + metricName + " " + value);
    },

    increment: function(metricName) {
        console.log("COUNTER_METRIC " + metricName);
    },

    error: function(metricName) {
        console.log("COUNTER_METRIC Error");
        console.log("COUNTER_METRIC " + metricName);
    }
}

This results in a set of log lines being automatically written to the CloudWatch log file. The example below shows a counter for the metric PlayLatestSermon, a timer for FeedFetchTime (taking 1,001ms) and a timer for FeedProcessingTime (taking 3,278ms).

2018-01-27T15:06:40.441Z	acd0af62-0373-11e8-94ab-53e50c1e53b5	COUNTER_METRIC PlayLatestSermon
2018-01-27T15:06:41.442Z	acd0af62-0373-11e8-94ab-53e50c1e53b5	TIMER_METRIC FeedFetchTime 1001
2018-01-27T15:06:44.720Z	acd0af62-0373-11e8-94ab-53e50c1e53b5	TIMER_METRIC FeedProcessingTime 3278

Now that this data is in the logs I can set up the filters.

Setting up the Metric Filters

Setting up the filters is pretty simple. Log into CloudWatch in the AWS portal, and click on ‘Logs’. Then click on the log group for your AWS Lambda Function and then click the blue ‘Create Metric Filter’ button.

 

 

You’ll then be prompted to enter the filter pattern. This is the pattern you want to match for in each log line. You can see from the log examples above that my log structure is in the format

DATETIME REQUESTID LOG_DATA

Since I’m logging two different types of metrics (timers and counters) my filters are pretty much one of two.

[datetime, requestId, level=TIMER_METRIC, metricName=FeedFetchTime, metricValue]
[datetime, requestId, level=COUNTER_METRIC, metricName=AMAZON.LoopOffIntent]

The first line looks for a log line starting with ‘TIMER_METRIC’ it then takes the next value as the metric name (for this example it’s the FeedFetchTime metric), then the value I write log. The second example is almost identical but the type of log is different, the metricName is different, and since it’s a counter I don’t emit a value.

You can test the filter pattern against sample log data but be warned, the drop down is in reverse order, so you can only test it against some of your oldest log data (not great if you’ve added new data that’s in your newer log files). To work around this you can copy/paste log data and test against that. Once you’re happy it’s matching log lines correctly you can configure the metric itself.

Here you can set a namespace to group your metrics together. The metric name (I copied the name that I emit to the logs to keep it simple). If you want to log the value from the logs then you can click the different quick-links below the value field (these will match the names you used in your filter pattern). For my example I used ‘metricValue’ but for counters I just leave this field blank since I don’t care about a specific value. When that’s done you can save your filter.

Graphing your new metrics

The metrics will normally take a few minutes to come through the CloudWatch metrics side. After a few minutes you should be able to click on the name of the namespace you configured in your metric filter and then find the metric you want. Once it’s graphed you can change the statistic type too. Some useful ones are

  • avg – The average value for the specified interval. Be careful with this stat, it’s nice to have but it can give you a false sense of security
  • sum – This calculates the sum of all of the values emitted in the time window
  • min – The lowest value emitted in the time window
  • max – The largest value emitted in the time window
  • sample count – The number of times the metric was emitted (useful for error counts etc, but make sure you don’t log error=0 or something similar since that will have a sample count of 1)
  • p99 / p90 / p50 etc – These are immensely valuable, these are percentile metrics. P99 shows you the values at the 99th percentile. In other words, they show the top 1% of values seen. Likewise P90 is the top 10% etc. These are great for timer metrics. The P99 execution time, shows you what the slowest (larger values = longer time) 1% of users experience. The P90 would show what the slowest 10% of users experience. This really show where users might be getting a much worse experience than you think if you just focus on the average values.

I suggest duplicating the same metric on a single graph and changing the statistic type to show avg, P90 and P99 on a single graph so you can get a feel for the average experience, but also what the top 1% and 10% of users experience too.

Creating a Dashboard

Once you’ve got data coming through to CloudWatch metrics I suggest creating a custom dashboard to show the data you care about in a single screen. Click on dashboard and pick a name and then click ‘add widget’ to add a new graph. You can expand graphs to take up different numbers of columns / rows but remember to click “save dashboard” whenever you modify something!

Building an Alexa podcast skill

Disclaimer: I’m a software engineer at Amazon, but everything expressed in this post is my own opinion. 

Photo by Andres Urena

What am I building?

Essentially I’m building a podcast skill for the Church I attend (Kingsgate Community Church). The aim is for people to be able to ask Alexa to play the latest sermon and have the device play back the audio. That way people can keep up-to-date if they missed a week at Church.

What I’m building would be easy to adapt (if you just wanted a generic podcast skill) and I’m keeping the code for the back-end on my public Github page.

Building the skill

Alexa skills are pretty simple to create if you use AWS (although you’re not restricted to this). There are two main parts to building a skill

  1. Configure the skill and define the voice commands (called an interaction model)
  2. Create a function (or API) that will provide the back-end functionality for your features

Configuring the skill

Creating a skill is pretty simple, you can visit the Alexa Developer Portal and click on “create a skill” where you’ll be asked some key questions.

  1. Skill type – For my podcast skill I’m using a custom skill so I can control exactly what commands and features I want to support (learn more about skill types)
  2. Name – This is what will be displayed in the skill store so it needs to be clear so that people can find your skill to enable it.
  3. Invocation Name – This is the word (or phrase) that people will say when they want to interact with your skill. If you used the invocation name “MySkill” then users would say “Alexa, ask MySkill…”. It doesn’t matter if someone else is using the same invocation name since users have to enable each skill they want to use. You should check the invocation name guidelines when deciding what to use.
  4. Global fields – The skill I’m building will support audio playback so I tick “yes” to use the audio player features.

Voice Commands (a.k.a the interaction model)

This is one of the parts that people struggle with when creating Alexa Skills. What you are really designing here is the interface for your voice application. You’re defining the voice commands (utterances) that users can say, and what commands (intents) these will be mapped to. If you want to capture a variable / parameter from the voice command, these can be configured into something called a ‘slot’. I find it easiest to think of intents like I would REST API requests. You configure the voice commands that will trigger an intent / request and then your skill handles each different intent just like you would handle different API requests if they came from a different button click command.

The developer portal has a great new tool for managing the interaction model. You can see from the screenshot on the right that I have a number of different intents defined. Some of these are built-in Alexa intents (like AMAZON.Stop) and I have some custom intents too. The main two intents I’ve configured are

  1. PlayLatestSermon – Used to fetch the latest sermon from the podcast RSS feed and start audio playback. Invoked by a user saying “Alexa, ask Kingsgate for the latest sermon”
  2. SermonInfoIntent – Gives details of the podcast title, presenter name, and publication date. Invoked y a customer saying “Alexa, ask Kingsgate for the latest sermon”

Adding an intent is as simple as clicking the add button, selecting the intent name, and then defining the utterances (voice commands) that users can say to trigger that intent. Remember: for custom intents the user will have to prefix your chosen utterance with “Alexa, ask <invocation name>…”, for built in Alexa intents (like stop, next, shuffle) the user doesn’t have to say “Alexa, ask…” first. It’s important to think about this as your user interface and pick utterances that users are likely to say without too much thought. You don’t want people to have to think about what to say so make sure you give lots of variations of phrasings. When you’re done you’ll need to click save and also build the model.

Creating the function

For my skill I’m using an AWS Lambda Function, which is a great way of publishing the code I want to run without having to worry about server instances, configuration or scaling etc.

Creating a skill is simple, just log into the AWS control panel in go to Lambda and then click create function. Pick the name, runtime (for my function I’m using Node.js 6.10 but you can use Go, C#, Java, or Python). Once you’ve created the function it will automatically be given access to a set of AWS resources (logging, events, DynamoDB etc). You’ll need to add a trigger (which defines the AWS components that can execute the Lambda function), since this is an Alexa skill I selected ‘Alexa Skills Kit’. If you click on the box you’ll be given the option to enter the ID of your Alexa skill (which is displayed in the Alexa Developer Portal). This gives an extra protection to make sure only your Alexa skill can execute the function. I’ve also given access to Cloudwatch Events, but I’ll cover this in another up-coming post about automated lambda monitoring).

The code for the lambda function is split into three main parts

  1. The entry point
    This sets up the Alexa SDK with your desired settings and also registers the handlers for different intents you support in different states. You can see the full entry code on Github.

    exports.handler = (event, context, callback) => {
      
        var alexa = Alexa.handler(event, context);
        alexa.appId = constants.appId; // Set your Skill ID here to make sure only your skill can execute this function
        alexa.dynamoDBTableName = constants.dynamoDbTable; // The DynamoDB table is used to store the state (of everything you set in this.attributes[] on a per-user basis).
    
        // Register the handlers you support
        alexa.registerHandlers(
            handlers.startModeIntentHandlers, 
            handlers.playModeIntentHandlers
        );
    
        alexa.execute();
    };
    
  2. The handlers
    The handler code defines each intent that is supported in different states “START_MODE”, “PLAY_MODE” etc. You can also have default code here for unhandled intent. This is a simplified version of the START_MODE handlers, you can see the full version on Github.

    var stateHandlers = {
    
        // Handlers for when the Skill is invoked (we're running in something called "START_MODE"
        startModeIntentHandlers : Alexa.CreateStateHandler(constants.states.START_MODE, {
    
            // This gets executed if we encounter an intent that we haven't defined
            'Unhandled': function() {
    
                var message = "Sorry, I didn't understand your request. Please say, play the latest sermon to listen to the latest sermon.";
                this.response.speak(message).listen(message);
                this.emit(":responseReady");
            },
    
            // This gets called when someone says "Alexa, open <skill name>"
            'LaunchRequest' : function () {
    
                var message = 'Welcome to the Kingsgate Community Church sermon player. You can say, play the latest sermon to listen to the latest sermon.';
                var reprompt = 'You can say, play the latest sermon to listen to the latest sermon.';
    
                this.response.speak(message).listen(reprompt);
                this.emit(':responseReady');
            },
      
            // This is when we get a PlayLatestSermon intent, normally because someone said "Alexa, ask <skill name> for the latest sermon"	
            'PlayLatestSermon' : function () {
                // Set the index to zero so that we're at the first/latest sermon entry
                this.attributes['index'] = 0;
                this.handler.state = constants.states.START_MODE;
    
                // Play the item
                controller.play.call(this);
            }
        
        }),
        ... rest of handler code
  3. The controller
    The controller is called by the handlers and takes care of interacting with the podcast RSS feed, calling the audio player to start playing the podcast on the device etc. The code is too long to show here so I’d suggest looking at the controller code in the Github repo.

Once you’re happy you can upload your skill code (or use the in-line editor). Since I have a few npm dependencies I zipped my function (and the node_modules folder) and uploaded it). You’ll also need to give the name of the function that should be executed when the function is called (for mine it’s index.handler).

You can then edit the configuration of your skill and point it to the ARN of your lambda function. You don’t have to publish the skill to start using it yourself, as long as your Alexa device is using the same account as your Alexa developer account, then you’ll be able to test the skill on your own device.

 Scroll to top