# Dataset Creation > Preprocess and create a dataset. After collecting data we need to turn it into a dataset. To do so there are a few processing steps that need to occur: 1. Extract labelled data from the rest of the unlabelled data. 2. Perform any preprocessing (like pre-calculating magnitudes and subtractive static gravity) 3. Condense into a single set 4. Sample windows of data from the set (ensuring balanced amounts of data) 5. Combine to create our dataset ## Extract Labelled Data Currently we have a large chunk of accelerometer data, containing both labelled (during experiment) and unlabelled data (outside of the experiment). First we can use the labelled data to identify the regions that are of interest (and dump the unlabelled data). To do that we can use the annotation events that we embedded. This is because the annotations contain: - The annotation label - The time of the annotation We can use the annotation "start" and "stop" markers as window start and stop. At the same time, we can also check that the data in each window is continuous. Data from the accelerometer is sensed at a consistent output sample rate. While streaming this data or recording, some of this data could have been dropped. Such artefacts must not taint the dataset and should be removed if present. An easy way to check this is by ensuring that adjacent samples are a maximum duration apart, and that if they are, split the window, keeping both either side provided they are a minimum duration. Here the gestures are about half a second long, so we can require that windows be a minimum of a second long. The `data_extract` tool can be used to simplify this. We need to supply: - `--annotations [ANNOTATION.csv]`: The `ANNOTATION.csv` file - `--data [ACC_2G.csv]`: The Accelerometer data file - `--label [labels.json]`: The `labels.json` file and specify that - Mode: `window` that we're using windows marked by "start"/"end" suffixes and finally to output the results to - `--output [output.csv]` Add in the Path and Infuse ID (without leading 0x) below: Altogether this creates: The same can be performed for the Gyroscope data. Note that the firmware configures the IMU to sample at 30Hz and 15Hz for the accelerometer and gyroscope respectively. However, the IMU used on the nRF54L15 Tag does not support these exact sample rates.
Therefore the driver selects the next closest sample rate which is 25Hz in both cases. ## Preprocessing The next step is to preprocess the data which is a bit more custom and depends on the scenario. Here are some useful things that will need to be done: - Merge the two accelerometer and gyroscope streams (which have two different sample rates) - Create a new column for Accelerometer Magnitude. - Create a new column for Gyroscope angular velocity. - Make sure that each sample is temporally adjacent (there is not gaps in data). Since the preprocessing is often more bespoke and scenario-dependent, it doesn't make sense to have a standard script. A sample is provided below. Your preferred coding AI tool is really useful to create and modify these sorts of scripts We'll start with the basics: Importing the data. We'll make use of command line arguments to specify the `--acc`, `--gyro`, and `--output` file paths. We'll also need the sample rates of each accelerometer and gyroscope stream (more on that later). ```python import argparse import pandas as pd if __name__ == "__main__": parser = argparse.ArgumentParser(description="Preprocess accelerometer and gyroscope data") parser.add_argument("--acc", type=str, required=True, help="Path to the accelerometer CSV file") parser.add_argument("--gyro", type=str, required=True, help="Path to the gyroscope CSV file") parser.add_argument("--output", type=str, required=True, help="Path to the output merged CSV file") parser.add_argument("--acc-rate", type=float, required=True, help="Sample rate of the accelerometer data in Hz") parser.add_argument("--gyro-rate", type=float, required=True, help="Sample rate of the gyroscope data in Hz") args = parser.parse_args() # Only require the gyroscope sample rate if gyroscope data is provided if args.gyro and not args.gyro_rate: raise ValueError("Gyroscope sample rate (--gyro-rate) must be provided if gyroscope data (--gyro) is provided.") # Load the accelerometer and gyroscope data acc_data = pd.read_csv(args.acc, parse_dates=['time'], date_format="ISO8601", dtype={"sample.x": "int16", "sample.y": "int16", "sample.z": "int16"}) gyro_data = pd.read_csv(args.gyro, parse_dates=['time'], date_format="ISO8601", dtype={"sample.x": "int16", "sample.y": "int16", "sample.z": "int16"}) if args.gyro else None # Rename axes to avoid confusion (sample prefix), but also avoid using '.' for Nordic Edge AI Lab acc_data.rename(columns={"sample.x": "acc_x", "sample.y": "acc_y", "sample.z": "acc_z"}, inplace=True) if gyro_data is not None: gyro_data.rename(columns={"sample.x": "gyro_x", "sample.y": "gyro_y", "sample.z": "gyro_z"}, inplace=True) ``` ### Merging the two streams Depending on the firmware, both accelerometer and gyroscope streams may or may not have different sample rates, we need to both resample then and align them in time before merging. There are numerous resampling algorithms, however whatever we pick needs to also run on the embedded system. As a result some of the fancier algorithms may not be suitable for real-time processing. To keep things simple, this demo will just use a Zero Order Hold (ZOH) filter. ```python # ... # Rename axes to avoid confusion (sample prefix), but also avoid using '.' for Nordic Edge AI Lab acc_data.rename(columns={"sample.x": "acc_x", "sample.y": "acc_y", "sample.z": "acc_z"}, inplace=True) if gyro_data is not None: gyro_data.rename(columns={"sample.x": "gyro_x", "sample.y": "gyro_y", "sample.z": "gyro_z"}, inplace=True) + # Find windows in both acc and gyro data with the same window ID (composed of label ID and sample ID columns) + if gyro_data is not None: + common_windows = set(acc_data['window_id']).intersection(set(gyro_data['window_id'])) + acc_data = acc_data[acc_data['window_id'].isin(common_windows)] + gyro_data = gyro_data[gyro_data['window_id'].isin(common_windows)] + + # Within the same window (using Window ID) Find the smallest offset between any two samples, and align the gyroscope data accordingly + if common_windows is not None and gyro_data is not None: + for window_id in common_windows: + acc_window = acc_data[acc_data['window_id'] == window_id].sort_values('time') + gyro_window = gyro_data[gyro_data['window_id'] == window_id].sort_values('time') + if not acc_window.empty and not gyro_window.empty: + time_offset = acc_window['time'].iloc[0] - gyro_window['time'].iloc[0] + gyro_data.loc[gyro_data['window_id'] == window_id, 'time'] += time_offset + # Remove samples where either gyro or acc data is outside the start/end of the other + gyro_data = gyro_data[~((gyro_data['window_id'] == window_id) & + ((gyro_data['time'] < acc_window['time'].iloc[0]) | + (gyro_data['time'] > acc_window['time'].iloc[-1])))] + acc_data = acc_data[~((acc_data['window_id'] == window_id) & + ((acc_data['time'] < gyro_window['time'].iloc[0]) | + (acc_data['time'] > gyro_window['time'].iloc[-1])))] + + # Resample the gyroscope data to match the accelerometer sample rate using Zero Order Hold (ZOH) + # merge_asof requires its `on` key to be globally sorted, not merely + # sorted within each window. `window_id` still keeps matches isolated + # to their corresponding sub-window via `by` below. + gyro_data = gyro_data.sort_values(['time', 'window_id']) + acc_data = acc_data.sort_values(['time', 'window_id']) + # Match each accelerometer sample with the most recent gyro sample in + # the same contiguous sub-window (zero-order hold). + gyro_columns = [column for column in ('gyro_x', 'gyro_y', 'gyro_z') + if column in gyro_data.columns] + gyro_data_resampled = pd.merge_asof( + acc_data[['window_id', 'time']], + gyro_data[['window_id', 'time'] + gyro_columns], + on='time', by='window_id', direction='backward', + ).drop(columns=['window_id', 'time']) + gyro_data_resampled.index = acc_data.index + + # Finally, merge the two streams + merged_data = acc_data.join(gyro_data_resampled) + merged_data.to_csv(args.output, index=False) + else: + merged_data = acc_data ``` ### Creating 1D columns & output the result After merging the accelerometer and gyroscope data, we want to create 1D columns for the magnitude and angular velocity. ```python # ... # Finally, merge the two streams merged_data = acc_data.join(gyro_data_resampled) merged_data.to_csv(args.output, index=False) else: merged_data = acc_data + # Create 1D columns for magnitude and angular velocity (as uint32 to prevent overflow) + merged_data['acc_magnitude'] = ((merged_data['acc_x'].astype('uint32')**2 + merged_data['acc_y'].astype('uint32')**2 + merged_data['acc_z'].astype('uint32')**2)**0.5).astype('uint32') + if gyro_data is not None: + merged_data['gyro_magnitude'] = ((merged_data['gyro_x'].astype('uint32')**2 + merged_data['gyro_y'].astype('uint32')**2 + merged_data['gyro_z'].astype('uint32')**2)**0.5).astype('uint32') + + metadata_columns = [column for column in ('label', 'label_id', 'window_id') + if column in merged_data.columns] + merged_data = merged_data[[column for column in merged_data.columns + if column not in metadata_columns] + metadata_columns] + merged_data.to_csv(args.output, index=False) ``` ### Contiguous Data Gaps in the data can lead to anomalies in data, especially when performing spectral analysis. Although this isn't likely to occur when using the data logger, streaming the live data over Bluetooth can sometimes have gaps. As a result, we can and should verify that data does not have any gaps and when they are present, split the window around said gap. This gets a bit more complicated when both accelerometer and gyroscopic data are present and at different sample rates. This is where we use the sample rates provided earlier, to determine the acceptable length between each sample. The additions are available below. ### End result The resultant scripts looks like this (and can be downloaded). Run the script to preprocess the data (accelerometer only): ```bash python preprocess_data.py --acc extracted_acc.csv --acc-rate 25 --output acc_only_windows.csv --min-window 1 ``` Or alternately include the Gyro data too: ```bash python preprocess_data.py --acc extracted_acc.csv --gyro extracted_gyr.csv --acc-rate 25 --gyro-rate 25 --output acc_gyro_windows.csv --min-window 1 ``` Here `--min-window` specifies the minimum number of consecutive samples required to form a valid window. This is used when there is a gap to ensure a subwindow is still large enough to be useful. ## Sampling We aren't done just yet though, since the data needs to be [balanced](https://developers.google.com/machine-learning/crash-course/overfitting/imbalanced-datasets) to prevent bias when data for each label isn't evenly distributed. We'll use a final script to perform sampling and balance the dataset. and run ```bash python balance_windows.py -i acc_only_windows.csv --num-samples 500 -o gesture_acc_dataset.csv --drop-non-numerical ``` The Nordic Edge AI Lab does not like non-numerical columns in the dataset, such as the label string and time columns. As a result, using the `--drop-non-numerical` flag is recommended. This script will determine how many samples are available under each label (e.g. printing out): ```text Input rows by label_id: label_id='2': 2968 row(s) label_id='3': 2542 row(s) label_id='4': 2696 row(s) label_id='5': 3240 row(s) label_id='6': 2744 row(s) label_id='7': 2614 row(s) label_id='8': 3908 row(s) Common target: 500 row(s) label_id='2': selected 1249 row(s) label_id='3': selected 1184 row(s) label_id='4': selected 658 row(s) label_id='5': selected 1177 row(s) label_id='6': selected 789 row(s) label_id='7': selected 725 row(s) label_id='8': selected 1509 row(s) ``` Select a number to use with `--num-samples` to maximize the number of samples for each label while maintaining balance. In this dataset, the fewest sample count is for label 3, containing 2500 samples. Therefore around 2500 would be a good choice. Finally the dataset is ready and we can get started with Nordic Edge AI Lab to train a model.