Software & Apps
Chenquan / Arkflow: High Performance Real Stream Processing Engine, which provides strong processing data capabilities, supporting multiple input / output sources.
English | Evalid
The High-Deferation Ream Stream Processing Engine, which provides strong data stream processing capabilities, supporting many input / output sources.
- Highly accomplished: Built in Rust and Tokio Async Runtime, offering excellent performance and low latency
- Many data sources: Support for Kafka, MQTT, HTTP, file, and other input / output sources
- Vigorous Procedures: Built-in SQL questions, JSON processing, protobuf encoding / decoding, batch processing, and other processors
- Unresponsive: Modular design, easy to provide to new input, output, and processor components
# Clone the repository
git clone https://github.com/chenquan/arkflow.git
cd arkflow
# Build the project
cargo build --release
# Run tests
cargo test
- Create a configuration file
config.yaml:
logging:
level: info
streams:
- input:
type: "generate"
context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
interval: 1s
batch_size: 10
pipeline:
thread_num: 4
processors:
- type: "json_to_arrow"
- type: "sql"
query: "SELECT * FROM flow WHERE value >= 10"
- type: "arrow_to_json"
output:
type: "stdout"
- Run Arkflow:
./target/release/arkflow --config config.yaml
Arkawlow uses YAML configuration files supporting the following main configuration items:
logging:
level: info # Log level: debug, info, warn, error
streams: # Stream definition list
- input: # Input configuration
# ...
pipeline: # Processing pipeline configuration
# ...
output: # Output configuration
# ...
Arkawlow supports multiple input sources:
- Kafka: Read data from Kafka subjects
- MQTT: Subscribe to messages from MQTT topics
- Http: Receive data through http
- file: Read data from files
- Generator: Create test data
- SQL: Question data from databases
Example:
input:
type: kafka
brokers:
- localhost:9092
topics:
- test-topic
consumer_group: test-group
client_id: arkflow
start_from_latest: true
ArkFlow provides multiple data processors:
- Json: Processing JSON data and change
- SQL: Data process using SQL questions
- Protobuf: Protobuf encoding / decoding
- Batch processing: Batch processing messages
Example:
pipeline:
thread_num: 4
processors:
- type: json_to_arrow
- type: sql
query: "SELECT * FROM flow WHERE value >= 10"
- type: arrow_to_json
Arkawlow supports multiple output targets:
- Kafka: Write data in subjects in Kafka
- MQTT: Pwali the messages of MQTT subjects
- Http: Send data by http
- file: Write data in files
- Standard output: Output data on console
Example:
output:
type: kafka
brokers:
- localhost:9092
topic: output-topic
client_id: arkflow-producer
streams:
- input:
type: kafka
brokers:
- localhost:9092
topics:
- test-topic
consumer_group: test-group
pipeline:
thread_num: 4
processors:
- type: json_to_arrow
- type: sql
query: "SELECT * FROM flow WHERE value > 100"
- type: arrow_to_json
output:
type: kafka
brokers:
- localhost:9092
topic: processed-topic
streams:
- input:
type: "generate"
context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
interval: 1ms
batch_size: 10000
pipeline:
thread_num: 4
processors:
- type: "json_to_arrow"
- type: "sql"
query: "SELECT count(*) FROM flow WHERE value >= 10 group by sensor"
- type: "arrow_to_json"
output:
type: "stdout"
Arkflow is licensed under the Apache 2.0 license.
https://opengraph.githubassets.com/c4edf8a33fa1baaab3a26045be8a00c4a7ebf96259813c68d66b5a669211545a/chenquan/arkflow
2025-03-14 03:58:00
