training-labs

Lab 6: Plotting Real Data

Both MATLAB and Python are excellent tools for plotting data. The choice of software and presentation format should be based on personal preferences and the characteristics of the team project. The ultimate objective is to achieve a highly effective visualization of the collected data.

Objective

By completing the sixth training lab, you will be able to:

load data saved from The Things Network
explore effective ways of presenting data

MATLAB or Python?

This lab attempts to support solutions that use MATLAB or Python. Both approaches are equivalent in terms of the objective. The language choice is for user comfort. Material common to both MATLAB and Python appears before the language-specific information.

What is JSON?

The lab 5 scripts saved your data into text files with a JSON object on every line. Each line represents a packet sent to The Things Network, and there are useful pieces of metadata such as the receipt time and the received signal strength from all gateways that successfully received the packet. JSON has only a few datatypes:

Number
String
JSON Object (similar to MATLAB struct or Python dictionary)
Array (MATLAB array or Python list)
Boolean
null

If we look at one line and press the enter key a few times to make parts of it more readable, we see this:

{"end_device_ids":
  { "device_id":"dev-test",
    "application_ids":{"application_id":"ac2456-v3-eval"},
    "dev_eui":"4AE3F8A1D400C6B5",
    "join_eui":"0000000000000001",
    "dev_addr":"260C4A9D"},
  "correlation_ids":[
    "as:up:01FXB1YXCTFEVYZ01W8NGGZ17C",
    "gs:conn:01FXB1Q5P7776HDSBHG5XN02RD",
    "gs:up:host:01FXB1Q5V071N34AT6Z0PMGYEC",
    "gs:uplink:01FXB1YX6670T8M387EHN508NK",
    "ns:uplink:01FXB1YX67ZZBE8EK7RX5YWWNT",
    "rpc:/ttn.lorawan.v3.GsNs/HandleUplink:01FXB1YX67G6SM6DJWAYDCR3R3",
    "rpc:/ttn.lorawan.v3.NsAs/HandleUplink:01FXB1YXCSJER82BK65WDB9EST"],
  "received_at":"2022-03-04T18:09:36.668759717Z",
  "uplink_message":
  { "session_key_id":"AX9WGmqHGOgokus3xx5qhA==",
    "f_port":1,
    "f_cnt":23,
    "frm_payload":"IQAAAEWblUM=",
    "decoded_payload":{"count":33,"temp":26.063043212890648},
    "rx_metadata":[
      {"gateway_ids":{"gateway_id":"ttn-ithaca-00-08-00-4a-3f-78","eui":"008000000001176B"},
      "time":"2022-03-04T18:09:36Z",
      "timestamp":252458507,
      "rssi":-78,
      "channel_rssi":-78,
      "snr":9.25,
      "location":{"latitude":39.0337215916,"longitude":-77.0183791462,"altitude":-1,"source":"SOURCE_REGISTRY"},
      "uplink_token":"CioKKAocdHRuLWl0aGFjYS0wMC0wOC0wMC00YS0zZi03OBIIAIAAAAABF2sQi+yweBoMCOCriZEGEKzJwdgBIPi1zL2sBw=="}
      ],
    "settings":{
      "data_rate":{"lora":{"bandwidth":125000,"spreading_factor":7}},
      "coding_rate":"4/5",
      "frequency":"904100000",
      "timestamp":252458507},
      "received_at":"2022-03-04T18:09:36.455926327Z",
      "consumed_airtime":"0.056576s",
      "version_ids":{"brand_id":"mcci","model_id":"catena4450","hardware_version":"_unknown_hw_version_","firmware_version":"0.3.0","band_id":"US_902_928"},
      "network_ids":{"net_id":"000013","tenant_id":"ttn","cluster_id":"ttn-nam1"}}}')

Of note: every packet is a JSON object with four keys. “received_at” is a string that represents the UTC time of receipt, and “uplink_message” is a JSON object that has a lot of useful information about the packet, registered device associated with that packet, and the gateways that received that packet. “end_device_ids” and “correlation_ids” have other information that may be useful in applications with many devices.

We want to save the “received_at” time to plot time series.

We also want to save our decoded payload. In the example packet, there is a temperature value that we want to save. In your decoder, you should replace “temp” by the name of your payload decoder field, like myint16, if you used a different name.

Making a Time Series from your Saved Data In MATLAB

If you have saved data using either the MATLAB or Python script (text file of JSON), you can put the following into a script (I called mine do_analysis.m) and run it to load the times and temperatures into variables called times and temps.

clear times;
clear temps;
fid = fopen("myData2.txt", "r");
line = fgets(fid);
times = [];
temps = [];
lineErrors = 0;
while line ~= -1 
    %disp(line);
    try
        js = jsondecode(line);
        nt = datetime(js.received_at, "InputFormat", 'yyyy-MM-dd''T''HH:mm:ss.SSSSSSSSSZ', "TimeZone", 'America/New_York');
        ntemp = js.uplink_message.decoded_payload.temp;
        %change 'temp' to what you call your decoded value, like 'RH' or 'myInteger'
        %you can build lists of whatever decoded values you have
        times = [times, nt];
        temps = [temps, ntemp];
        % the above two lines perform list concatenation.
        % we append a new value to existing lists
    catch ME
        %fprintf("Error with line:\n%s\n", line);
        lineErrors  = lineErrors +  1;
    end
    
    line = fgets(fid);
end

I was able to plot my data with

plot(times, temps);

but you will probably want to do more analysis than I did. You should also make a nicer looking plot than I made with that line of plot().

Making a Time Series from your Saved Data In Python

#!/usr/bin/env python3 -i
import json
import matplotlib.pyplot as plt
from datetime import date
from dateutil.parser import parse
import pytz


times = []
temps = []
with open("myData2.txt") as f:
  for line in f:
    try:
      js = json.loads(line)
      ntime = parse(js["received_at"]).astimezone(pytz.timezone('US/Eastern'))
      ntemp = js['uplink_message']['decoded_payload']['temp']
      #change 'temp' to what you call your decoded value, like 'RH' or 'myInteger'
      #you can build lists of whatever decoded values you have
      times.append(ntime)
      temps.append(ntemp)
    except:
      pass

I was able to run my code on Linux (or MacOS) at the command line as

acoy@hplaptop0:~/Documents/ECE_4950/mqtt_matlab$ python3 -i plot_python.py

Python on Windows uses the py launcher on Powershell or the cmd.exe prompt.

Powershell>> py -i plot_python.py

That program loads the times into a list called times, and loads the corresponding temperatures into a list called temps. Using python3 -i leaves me at a Python prompt so that I can experiment with analysis interactively. You can always put your analysis and plotting code into a script and not run interactively.

I was able to plot my data with

plt.plot(times, temps);plt.show()

but you will probably want to do more analysis than I did. You should also make a nicer looking plot than I made with that line of plot().

I was able to exit with Ctrl-D on MacOS and Linux, and Ctrl-Z then enter on Windows. You can also use the quit() function to exit Python.

Deliverable

Please do some cool analysis of some saved sensor data that you (and perhaps others) sent over TTN. Please present a brief report of that analysis as a pdf document. The report should have sections for the following:

Introduction
Plots of your analysis
Conclusion

Some examples of analysis are:

See how well your thermistor agrees with those of other students. You may share data for that collaboration. This will probably require finding a way to see which time-temperature points are “close enough” to each other in time.
Compare the variation of temperature in a room over time. Is the room temperature more variable at one time than another?
There may be other interesting things to look at.

The objective is to get some practice with studying and presenting data that you sent over LoRaWAN and the internet.

As always, please submit whatever code you used to send data, decode data, and make your visuals. The Canvas submission will be a .zip file with your .pdf report and any .ino, .c, .h, or other code files except your keys.h.