IoT Apache Kafka & MQTT

IoT Challenges

• Real time data processing
• Large volume of data
• Time to market
• Accuracy

Why Kafka

Kafka-native options to note for MQTT integration beyond Kafka client APIs

• Kafka Connect source and sink connectors, which integrate with MQTT brokers in both directions
• Confluent MQTT Proxy, which ingests data from IoT devices without needing a MQTT broker
• Confluent REST Proxy for a simple but powerful HTTP-based integration

Use cases for IoT technologies and an event streaming platform

Few use case Kafka & MQTT are already used in many IoT deployments, both in Consumer IoT and Industrial IoT (IIoT). 

• Connected car infrastructure: cars communicate with each other and the remote datacenter or cloud to perform real-time traffic recommendations, prediction maintenance, or personalized services.

   Example: Audi (https://www.confluent.io/kafka-summit-london18/keynote-fast-cars-in-a-streaming-world-reimagining-transportation-at-audi/)

• Smart cities and smart homes: Buildings, traffic lights, parking lots, and many other things are connected to each other in order to enable greater efficiency and provide a more comfortable lifestyle. Energy providers connect houses to buy or sell their own solar energy and provide additional digital services.

          Example: E.ON(https://de.slideshare.net/ConfluentInc/energy-data-streaming-decentral-energy-distribution-in-a-digital-world)

• Smart retail and customer 360: Real-time integration between mobile apps of customers and backend services like CRMs, loyalty systems, geolocation, and weather information creates a context-specific customer view and allows for better cross-selling, promotions, and other customer-facing services.

               Example: Target (https://www.confluent.io/kafka-summit-nyc17/simplifying-omni-channel-retail-at-scale/)

• Intelligent manufacturing: Industrial companies integrate machines and robots to optimize their business processes and reduce costs, such as scrapping parts early or predictive maintenance to replace machine parts before they break. Digital services and subscriptions are provided to customers instead of just selling them products.

          Example: Severstal

IoT standards MQTT, OPC UA, Siemens S7, and PROFINET

There are many IoT standards and technologies available on the market. If we had to choose, these are the most common options for implementing IoT integrations:

• Proprietary interfaces: especially in Industrial IoT (IIoT), this is the most common scenario. Machines provide a large number of usually closed and incompatible protocols in a proprietary format. Examples are S7, PROFINET, Modbus, or an automated dispatch system (ADS). Supervisory control and data acquisition (SCADA) is often used to control and monitor these systems.
• OPC UA: this is an open and cross-platform, machine-to-machine communication protocol for industrial automation. Every device must be retrofitted with the ability to speak a new protocol and use a common client to speak with these devices. License costs and modification of the existing hardware are required to enable OPC UA.
• PLC4X: As an Apache framework, it provides a unified API by implementing drivers (similar to JDBC for relational databases) for communicating with most industrial controllers in the protocols they natively understand. No license costs or hardware modifications are required.
• MQTT: This is built on top of TCP/IP for constrained devices and unreliable networks, applying to many (open source) broker implementations and many client libraries. It contains IoT-specific features for bad network/connectivity, and is widely used (mostly in IoT, but also in web and mobile apps via MQTT over WebSockets).

IoT Requirements

• Scalable data movement and processing: handles backpressure and can process increasing throughput
• Agile development and loose coupling: different sources and sinks should be their own decoupled domains. Different teams can develop, maintain, and change integration to devices and machines without being dependent on other sources or the sink systems that process and analyze the data. Microservices, Apache Kafka, and Domain-Driven Design (DDD) covers this in more detail.
• Innovative development: new and different technologies and concepts can be used depending on the flexibility and requirements of a specific use case. For instance, one application might already send data to an MQTT broker so that you can consume from there while another project does not use an MQTT broker at all, and you just want to push the data into the event streaming platform directly for further processing.

IoT Challenges 

• Complex infrastructure and operations that often cannot be changed—despite the need to integrate with existing machines, you are unable g to add code to the machine itself easily
• Integration with many different technologies like MQTT or OPC Unified Architecture (OPC UA) while also adhering to legacy and proprietary standards
• Unstable communication due to bad IoT networks, resulting in high cost and investment in the edge

Kafka: Event Streaming Platform

Apache Kafka is an event streaming platform that combines messaging, storage, and processing of data to build highly scalable, reliable, secure, and real-time infrastructure. 

Pros

• Stream processing, not just queuing
• High throughput
• Large scale
• High availability
• Long-term storage and buffering
• Reprocessing of events
• Good integration with the rest of the enterprise
• Hybrid, multi-cloud, and global deployments

Cons

• Not built for tens of thousands of connections
• Requires a stable network and solid infrastructure
• Lacks IoT-specific features like Keep Alive and Last Will and Testament

MQTT connectors

• The MQTT connectors are independent of a specific MQTT broker implementation.
• The MQTT broker is persistent and provides MQTT-specific features. 

MQTT Proxy

• MQTT broker is just added complexity, cost, and operational overhead.
• Confluent MQTT Proxy delivers a Kafka-native MQTT proxy that allows organizations to eliminate the additional cost and lag of intermediate MQTT brokers.
• MQTT Proxy accesses, combines, and guarantees that IoT data flows into the business without adding additional layers of complexity.
• MQTT Proxy is horizontally scalable, consumes push data from IoT devices, and forwards it to Kafka brokers with low latency. 
• The Kafka broker is the source of truth responsible for persistence, high availability, and reliability of the IoT data. 

Article By:

“Rajendra Pachouri Hands-on Software Engineering, API, Cloud, IoT & DevOps experience, and a proven track record of architecting and delivering reliable and scalable systems in a variety of areas such as e-commerce, payments, security, home automation, financial systems, messaging services, EDI, EAI Microservices”.

Comments

comments

Tags: cloud, IoT, Kakfa, MQTT