This article demonstrates an AI system that can automatically classify 160 different shortwave signal modes. This covers most of the radio frequency signals present in the HF spectrum. It uses a deep neural network trained on a large-scale dataset of example synthetic and real RF signals. The system achieves a classification accuracy of 90 % and a top3-accuracy of 95 % in real-world operation with a signal observation time of only 1 second.
Introduction to Automatic Identification of RF Signals
Automatic signal classification or identification determines the type (also called mode) of an RF signal. This is useful for several purposes such as
- Spectrum monitoring and surveillance
- Support of dynamic spectrum access in cognitive radios
- Identification of digital transmission modes in amateur radio
The input to the classifier is a portion of an IQ radio signal, that contains a particular mode. The classifier uses an AI model, a deep neural network, to detect the type of signal present. The neural network in this article is specifically designed for the identification of shortwave signals, such as Morse code, single-sideband voice, HFDL or AM broadcast. A full list of supported signal types can be found below.
Shortwave Signals
The shortwave or HF spectrum ranges from 3 to 30 MHz and is characterised by worldwide signal propagation. This is because shortwave radio waves are reflected at ionospheric layers in the atmosphere, which enables beyond line-of-sight communications. The use of the shortwave spectrum has some unique and attractive advantages for radio operators:
- Worldwide communication can be established with little effort (a transceiver and antenna are all that is required)
- No additional intrastructure such as cables, satellites or other backbones are required
- Communication links are difficult to suppress or block
Accordingly, a shortwave receiver typically receives many different signals from all over the world. As a result, a wide range of different modes can be present in a shortwave receiver. Some typical users of the shortwave bands are:
- Broadcasting (e.g. AM or DRM)
- Amateur radio (wide variety of analog and digital modes)
- Aviation (e.g. long-range aircraft communications, HFDL)
- Weather services (e.g. Wefax, RTTY weather data)
- Embassies
- Intelligence, military and security authorities
- Radar (civil and military) and ionosondes
An overview of common shortwave signals can be found in my compact overview of HF signals: The World of Shortwave Signals or at sigidwiki.com.
Identification of RF Signals with Deep Learning
In recent years it has been shown that many classification problems can be solved well with AI and machine learning approaches based on neural networks. This includes the related task of automatic modulation classification (AMC) for RF communication signals (e.g. https://arxiv.org/pdf/1602.04105.pdf). However, pure AMC can only detect textbook modulation types, such as BPSK, QPSK or FSK, and the number of output classes considered in many publications is quite small (mostly around 20 signal types). However, also for mode identification neural networks are a promising and powerful approach.
In general, a neural network processes signals differently from classical algorithms. Classical algorithms are step-by-step procedures, usually designed “by hand” by a developer who has some understanding of the underlying problem. In contrast, neural networks learn from masses of example data in a training process. In principle, neural networks are flexible mathematical models with an extremely large number of parameters. During training, the neural network adapts its parameters to the data and thereby learns to implement a functionality to map the input data to the correct output classes. Neural networks are therefore advantageous for complex problems where classical algorithms are not available.
Neural Network for RF Signal Identification
Here, a convolutional neural network (CNN) is used for the identification of RF signals. The following figure sketches the principle of a general CNN. The IQ signal passes through several processing layers. Each layer progressively modifies the signal to extract characteristic features and properties. In the final layer, the network indicates the most likely output class.
Typical processing layers in a CNN include:
- Convolutional Layer: This layer basically works like a bank of filters. The filters are similar to FIR filters, but typically include a non-linear output function. Different filters are applied to the signal in parallel. The filter coefficients are not predefined, but are learnt during training.
- Pooling Layers: This layer reduces the length of the signal by combining multiple samples (e.g. 2) into a single sample. This operation is similar to decimation and results in a shorter signal.
- Fully Connected Layer: This layer combines the filtered and pooled signals into several single output neurons. It is often found as the output layer at the end of the neural network, where each neuron corresponds to an output class. Each of the signal samples is weighted for each of the output classes. The weights are learnt during training.
The neural network in this work consists of an arrangement of 28 layers including convolutional pooling and fully connected layers.
RF Training Data
The training data is based on synthetic and high quality real-world signals from various sources, such as sigidwiki.com and many more, including own recordings and software generated signals. These signals are artificially distorted to provide diverse and realistic training data. This enables the neural network to learn to ignore typical distortions, like noise, fading or frequency offsets, that are present in an RF system. Useful distortions and details on data augmentation can be found in my article on RF dataset generation.
The dataset consists of complex IQ signals with a sampling rate of 4 kHz, thus covering a bandwidth of approximately 4 kHz, which is typical of most shoftwave signals. The length of each training signal is 1 second. The complete training dataset for this system contains a total of 1.2 million RF signals. The training dataset covers the following 160 signal modes:
Real-World Test Data
In machine learning, trained models are often evaluated against test data from the same underlying (and often synthetic) dataset. However, for deployment and real-world operation, it is important to test the neural network on real-world data. For this purpose a large test dataset of shortwave signals has been collected, that covers different real-world scenarios.
The test data consists of recordings from different SDR receiver hardware at different locations, daytime, season and operating frequency. These variations result in a highly diverse set of test data that provides a good representation of practical operation. The test data also includes information on the signal-to-noise ratio, which allows the evaluation of accuracy versus SNR.
| Tested Modes | 143 (most frequently occurring) out of 160 |
| SDR Hardware | Kiwi SDR, Airspy, SDR Play, Twente WebSDR, Elad FDM-S3 |
| Locations | Worldwide |
| Frequencies | across 3-30 MHz |
| Daytime and Season | All seasons and varying daytime |
| SNR | -10 to 25 dB |
| Recording Duration | >35 hours total |
Results
The trained neural network is now tested on real-world data. For that purpose the real-world data is fed into the neural network in 1 second chunks. The network then predicts the estimated signal class for each chunk. The accuracy measure indicates the success rate of the classifier, i.e. the ratio Number of correct predictions / Number of total predictions. In addition, the top-3 accuracy is reported. For the top-3 accuracy the network outputs the three most likely classes. If one of these matches the true class, it is counted as ‘correct’.
The results show very good accuracy values, which reach in average around 90 % for high SNR values. This means that in 9 of 10 cases the classifier selected the correct mode out of 160 possible, based on just one second of observation. The top-3 accuracy is at approximately 95 %. In addition, the classifier is robust to noise and achieves good accuracy also for low SNR signals.
The following plots show the results for a selection of some common shortwave signal types. Here, some variance over the different class types can be observed: While some modes may achieve accuracies below 80 %, others are identified with almost 100 % accuracy (at sufficiently large SNR values).
The confusion matrix below gives a more detailed picture of the classification and shows the output of the neural network dependent on the actual (ground truth) class. The rows and columns correspond to the true and predicted signal classes, respectively. The entries on the main diagonal represent correctly classified cases.
Why does it not always achieve 100 %?
Although the neural network performs very well on practical RF data, for some signals the classification accuracy does not reach 100 % even under high SNR conditions. There are several possible reasons for this:
- The observation time is only one second, so the system must decide based on a very short excerpt of the signal. This can be a problem for similar signal types with low-baud rate that may not be distinguishable in this short time, like Dominoex 8 vs MFSK-8 vs Thor-8.
- Some special digital modes are very similar and generally difficult to distinguish, e.g. MFSK-32 vs Olivia 16/500 (both 16-MFSK, 31 Bd, same shaping)
- High SNR does not necessarily mean “good signal quality”, e.g. even high SNR signals can suffer from severe fading or interference, which may impact the classification accuracy.
- For few signal classes that are proprietary and occur only rarely, good training data is hard to obtain.
Automatic identification of RF signals in the shortwave spectrum and at other frequencies is an exciting topic, but it is a niche in the SDR community. If you find this interesting or have any comments or suggestions, please contact me at research@panoradio-sdr.de
