3.3.1. Measure The Performance of the IoT Applications¶
| ID | Priority | Best Practice |
|---|---|---|
| BP 3.3.1.1 | Highly Recommended | Analyze the runtime performances of your application |
| BP 3.3.1.2 | Required | Add timestamps to each message published |
| BP 3.3.1.3 | Required | Load test your IoT applications |
Architecture Notes - BP 3.3.1.1 - Analyze the runtime performances of your application¶
Application performances at runtime are different from what you can observe in a controlled test environment. Actively analyzing the performances of your application based on device health, network latency, and payload size provides insight on how to obtain performance improvements. By using different types of metrics, the health of each device in a multi-device setting can be obtained.
Recommendation 3.3.1.1.1 - Analyzing anomaly as well as setting up a security profile will allow detection of anomalies along with having granular control of the device should changes be necessary
- Measurement of changes in connection patterns of devices may indicate some devices having a jittery network connection
- Measurement of device-side timestamps from multiple devices in comparison to arrival times on the cloud-side may indicate local network latency or additional hops in device path
- AWS IoT Device Defender Detect
Architecture Notes - BP 3.3.1.2 - Add timestamps to each message published¶
Timestamps helps in determining delays that might incur during the transmission of a message from the device to the application. Timestamps can be associated to the message and to fields contained in the message. If a timestamp is included, the sent timestamp, along with the sensor or event data, is recorded on the cloud-side.
Recommendation 3.3.1.2.1 - Add timestamp on the server side
- If the devices lack the capability to add timestamps to the messages, consider using server-side features to enrich the messages with timestamps corresponding to the reception of the message.
- For example, AWS IoT Rules SQL language provides a timestamp() function to generate a timestamp at the reception of the message.
Recommendation 3.3.1.2.2 - Have a reliable time source on the device
- Without a reliable time source, the timestamp can only be used relative to the specific device.
- For example:
- Devices should use the Network Time Protocol to obtain a reliable time when connected
- Real Time Clock devices can be used to maintain an accurate time while the device lacks connectivity
- Depending on the application, timestamps can be added at the message level or at the single payload field level. Delta encoding can be used to reduce the size of the message when multiple timestamps are included. Choosing the right approach is a trade-off between accuracy, energy efficiency, and payload size.
- Developer Guide - timestamp()
- Time series compression algorithms
- Delta encoding
Architecture Notes - BP 3.3.1.3 - Load test your IoT applications¶
Applications can be complex and have multiple dependencies. Testing the application under load help identify problems before going into production. Load testing your IoT applications ensures that you understand the cloud-side performance characteristics and failure modes of your IoT architecture. Testing helps you understand how your application architecture operates under load, identify any performance bottlenecks, and apply mitigating strategies prior to releasing changes to your production systems.
Recommendation 3.3.1.3.1 - Simulate the real device behavior
- A device simulator should implement the device behavior as close as possible. Do not only test message publishing, but also: connections, reconnections, subscriptions, enrollment and other environmental disruptive events. Start testing at a lower load, progressively increasing to over 100%.
- For example:
- Load test at a low percent of your estimated total device fleet, for example, 10%.
- Evaluate the performance of your application using operational dashboards created to measure end-to-end delivery of device telemetry data and automated device commands.
- Make any necessary changes to the application architecture to achieve desired performance goals.
- Iterate these steps increasing the load.
- IoT Device Simulator
- From testing to scaling