Software & Apps

Chenquan / Arkflow: High Performance Real Stream Processing Engine, which provides strong processing data capabilities, supporting multiple input / output sources.

English | Evalid

! ️Not prepare production, do not use in a production environment! ️

bet
can

The High-Deferation Ream Stream Processing Engine, which provides strong data stream processing capabilities, supporting many input / output sources.

  • Highly accomplished: Built in Rust and Tokio Async Runtime, offering excellent performance and low latency
  • Many data sources: Support for Kafka, MQTT, HTTP, file, and other input / output sources
  • Vigorous Procedures: Built-in SQL questions, JSON processing, protobuf encoding / decoding, batch processing, and other processors
  • Unresponsive: Modular design, easy to provide to new input, output, and processor components
# Clone the repository
git clone https://github.com/chenquan/arkflow.git
cd arkflow

# Build the project
cargo build --release

# Run tests
cargo test
  1. Create a configuration file config.yaml:
logging:
  level: info
streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
      interval: 1s
      batch_size: 10

    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT * FROM flow WHERE value >= 10"
        - type: "arrow_to_json"

    output:
      type: "stdout"
  1. Run Arkflow:
./target/release/arkflow --config config.yaml

Arkawlow uses YAML configuration files supporting the following main configuration items:

logging:
  level: info  # Log level: debug, info, warn, error

streams:       # Stream definition list
  - input:      # Input configuration
      # ...
    pipeline:   # Processing pipeline configuration
      # ...
    output:     # Output configuration
      # ...

Arkawlow supports multiple input sources:

  • Kafka: Read data from Kafka subjects
  • MQTT: Subscribe to messages from MQTT topics
  • Http: Receive data through http
  • file: Read data from files
  • Generator: Create test data
  • SQL: Question data from databases

Example:

input:
  type: kafka
  brokers:
    - localhost:9092
  topics:
    - test-topic
  consumer_group: test-group
  client_id: arkflow
  start_from_latest: true

ArkFlow provides multiple data processors:

  • Json: Processing JSON data and change
  • SQL: Data process using SQL questions
  • Protobuf: Protobuf encoding / decoding
  • Batch processing: Batch processing messages

Example:

pipeline:
  thread_num: 4
  processors:
    - type: json_to_arrow
    - type: sql
      query: "SELECT * FROM flow WHERE value >= 10"
    - type: arrow_to_json

Arkawlow supports multiple output targets:

  • Kafka: Write data in subjects in Kafka
  • MQTT: Pwali the messages of MQTT subjects
  • Http: Send data by http
  • file: Write data in files
  • Standard output: Output data on console

Example:

output:
  type: kafka
  brokers:
    - localhost:9092
  topic: output-topic
  client_id: arkflow-producer

Kafka to Kafka Data Processing

streams:
  - input:
      type: kafka
      brokers:
        - localhost:9092
      topics:
        - test-topic
      consumer_group: test-group

    pipeline:
      thread_num: 4
      processors:
        - type: json_to_arrow
        - type: sql
          query: "SELECT * FROM flow WHERE value > 100"
        - type: arrow_to_json

    output:
      type: kafka
      brokers:
        - localhost:9092
      topic: processed-topic

Create testing data and process

streams:
  - input:
      type: "generate"
      context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
      interval: 1ms
      batch_size: 10000

    pipeline:
      thread_num: 4
      processors:
        - type: "json_to_arrow"
        - type: "sql"
          query: "SELECT count(*) FROM flow WHERE value >= 10 group by sensor"
        - type: "arrow_to_json"

    output:
      type: "stdout"

Arkflow is licensed under the Apache 2.0 license.


https://opengraph.githubassets.com/c4edf8a33fa1baaab3a26045be8a00c4a7ebf96259813c68d66b5a669211545a/chenquan/arkflow

2025-03-14 03:58:00

Related Articles

Leave a Reply

Your email address will not be published. Required fields are marked *

Back to top button