Industrial and IoT Data: Transforming Field Streams into Reliable and Legally Admissible AI Data

The Industrial Internet of Things (IIoT) generates petabytes of measurements daily from sensors, programmable logic controllers (PLCs), and embedded systems. To support AI, these streams must be reliable, traceable and admissible in an audit or legal proceeding. This article outlines the technical, organisational and regulatory requirements for turning field signals into certified datasets for predictive models, decision-making dashboards or compliance procedures. It presents a reference architecture and governance, quality and security practices, then illustrates them with an industrial predictive-maintenance scenario.

Introduction – Verifying industrial data and decisions

'Smart' factories rely on massive data collection to optimise production lines, reduce downtime and anticipate failures using AI. However, the mere abundance of data is not enough: when a model predicts a failure or a statistical analysis justifies a line shutdown, the results must be verifiable, reproducible and defensible before stakeholders – management, internal auditors, regulatory authorities or courts.

This requirement for enforceability stems from several factors:

• the proliferation of legal requirements (GDPR, NIS2, IEC 62443) which mandate full traceability of data processing;
• contractual and insurance requirements where the client or insurer demands proof that automated decisions are based on reliable datasets;
• the need for industrial operators to justify their AI investments to shareholders by demonstrating a measurable return on investment.

The challenge therefore lies in architecting, securing and governing the pipeline from the sensor to the AI model, while integrating quality and compliance controls capable of withstanding rigorous scrutiny. The remainder of this article details the levers to be activated to achieve this.

1. Industrial context: IoT uses and challenges

These transformations pave the way for AI: supervised or unsupervised learning algorithms require large volumes of labelled, synchronised, and deduplicated data. However, transitioning from 'field noise' to actionable datasets requires a robust governance foundation.

2 From raw measurement to structured data: the collection chain

• Sensors and Acquisition – Sensors (temperature, vibration, current) convert physical phenomena into electrical signals. Sensor precision, calibration, and sampling frequency are the primary drivers of quality; poor calibration propagates through the entire pipeline.

• Edge Computing – Edge gateways (e.g., NVIDIA Jetson, Siemens Edge‑IIoT) perform local preprocessing: anti-aliasing filtering, temporal aggregation, and basic anomaly detection. This stage reduces latency and the volume of data transmitted to the cloud, while providing an initial layer of resilience (continuity even in the event of connectivity loss).

• Secure Transport – Industrial protocols (OPC UA, MQTT‑TLS, DDS) ensure the integrity and confidentiality of data flows. TLS 1.3 encryption with X.509 mutual authentication is recommended to prevent man-in-the-middle attacks.

• Data Lake Ingestion – Cloud services (Azure Data Lake, AWS S3, Google Cloud Storage) store raw data as immutable objects. The use of compressed columnar formats (Parquet, ORC) facilitates subsequent reading and optimises storage costs.

• Metadata and Cataloguing – A data catalogue (AWS Glue, Azure Purview) records the schema, origin, owner, and timestamps. This metadata layer is essential for legal traceability: every dataset becomes 'audit-ready'.

Strict adherence to these steps creates a chain of integrity where each link can be independently verified.

3 Industrial Data Quality: Assurance Criteria and Methods

Organisations often adopt the DQI (Data Quality Initiative) framework or ISO 8000 recommendations to formalise these controls. Observability tools (Datadog, Grafana Loki) enable real-time visualisation of pipeline health and automate remediation (re-ingestion, recalibration).

Each quality rule must be documented, versioned and assigned to an owner. In the event of a dispute, that person can show that the data was processed according to the established requirements.

4 Governance and traceability: ensuring data enforceability

4.1 Governance framework

4.2 Legal Traceability

• Immutable logging – Each transformation (filtering, aggregation, anonymisation) generates a signed record: hash(prev_record) + operation. Solutions based on private blockchains or cloud logging services (Azure Immutable Blob Storage) ensure inalterability.
• Retention – GDPR mandates the retention of compliance evidence for 5 to 10 years depending on the sector; IEC 62443 standards recommend a minimum of 7 years for industrial logs.
• Controlled Access – RBAC (Role-Based Access Control) and ABAC (Attribute-Based Access Control) policies restrict read/write access to authorised personnel only, with full access auditing.

These mechanisms allow an external auditor to trace data from the sensor to the AI-generated output and establish its admissibility as evidence.

5 Reference Technical Architecture

Read the diagram notation

flowchart TD
    subgraph Edge["Nœuds Edge"]
        C1[Capteurs] -->|Signal analogique| EP[Passerelle Edge]
        EP -->|Filtrage & agrégation| B1[Broker MQTT‑TLS]
    end

    subgraph Fog["Couche Fog"]
        B1 -->|Transport sécurisé| S1["Service d’ingestion (Kafka)"]
        S1 -->|Topic raw_data| L1["Data Lake (S3/ADLS)"]
        S1 -->|Topic metadata| Cata[Catalogue & Méta‑données]
    end

    subgraph Cloud["Plateforme Cloud"]
        L1 -->|Parquet| DWH["Entrepôt (Snowflake/BigQuery)"]
        DWH -->|SQL/ML pipelines| ML[Entraînement modèle]
        ML -->|Modèle| API[Service d’inférence]
        API -->|Résultat IA| App[Application métier]
    end

    style Edge fill:#f9f,stroke:#333,stroke-width:2px
    style Fog fill:#bbf,stroke:#333,stroke-width:2px
    style Cloud fill:#bfb,stroke:#333,stroke-width:2px

Main Components

This architecture separates latency concerns (edge) from scalability (cloud), while ensuring transactional consistency through immutable logs and a centralised catalogue.

6 Data pipeline security & resilience

The implementation of Zero Trust at every level – strong device identity, network micro-segmentation and continuous behavioural verification – is now considered the recommended best practice by ANSSI and NIST [3].

7 Preparing industrial data for AI

• Ingestion & cleaning – Use Spark Structured Streaming or Flink to transform raw streams into fact tables, applying validation rules (e.g., out-of-bounds values → rejection).
• Labeling – Machine downtime events are often manually annotated by maintenance teams; the integration of human-in-the-loop tools (e.g., Scale AI, Labelbox) ensures label quality and creates an audit trail.
• Feature engineering – Calculation of rolling averages, derivatives (vibration RMS), and frequency transformations (FFT) directly at the stream processing level. These features are versioned within the catalogue.
• Imbalance management – As failures are rare, apply oversampling techniques (SMOTE) or synthetic generation via industrial GANs, while maintaining full traceability of the synthetic data used.
• Training & validation – Separate datasets using temporal splits to prevent temporal leakage; log hyperparameters and metrics (AUC, F1) in an MLOps registry.

Adherence to these steps ensures that the AI model is based on certified data, with its provenance and transformation fully documented.

8 Regulatory framework & compliance

Compliance with these frameworks requires risk mapping (ISO 31000), the implementation of a record of processing activities (GDPR) including IoT flows, as well as periodic review processes to ensure alignment between legal requirements and operational practices.

9 Use case: predictive maintenance in a steel production plant

Context

A steelworks employing 2,000 people aims to reduce unplanned downtime of electric arc furnaces. Sensors installed on the equipment generate large volumes of time-series data. Their quality must be checked before the data enters a predictive-maintenance model. The challenge is to turn noisy raw signals into reliable indicators for critical operational decisions.