GitHub – oldmoe/litestack

All your data infrastructure, in one gem!

Litestack is a Ruby gem that provides Ruby and Ruby on Rails applications with an all-in-one solution for web application data infrastructure. It takes advantage of the power and flexibility of SQLite to provide a full SQL database, a fast cache, a robust work queue, a reliable message broker, a full text search engine and a metrics platform all in one package.

Compared to conventional methods that require separate servers and databases, Litestack offers better performance, efficiency, ease of use, and cost savings. Its embedded database and cache reduce memory and CPU usage, while its simple interface streamlines the development process. Overall, Litestack sets a new standard for web application development and is an excellent choice for those who demand speed, efficiency, and simplicity.

You can read more about why litestack might be a good choice for your next web application HEREyou might also be interested in litestack benchmarks.

With litestack you only need to add one gem to your app that replaces many other gems and services, for example, a standard Rails app using litestack will not need the following services:

• Database Server (eg PostgreSQL, MySQL)
• Cache Server (eg Redis, Memcached)
• Job Processor (eg Sidekiq, Goodjob)
• Pubsub Server (eg Redis, PostgreSQL)
• Fulltext Search Server (eg Elasticsearch, Meilisearch)

To make it more efficient, litestack can detect the presence of Fiber based IO frameworks such as Async (for example if you use the Falcon web server) or Polyphony. Then it moves its background workers for caches and queues to threads (using the semantics of the current framework). This is done transparently and generally leads to lower CPU and memory usage.

Add the litestack gem line in your application’s Gemfile:

To configure a Rails application to run the entire litestack, run:

$ rails generate litestack:install

Litestack currently offers six main components

• litedb
• litcache
• litejob
• read
• litesearch
• litemetric

litedb is a wrapper around SQLite3, which offers a better default configuration tuned for concurrency and performance. Out of the box, liteb works smoothly between multiple processes with no database locking errors. liteb can be used in several ways, including:

litedb can be used like the SQLite3 gem, because litedb inherits from SQLite3

require 'litestack'
db = Litedb.new(path_to_db)
db.execute("create table users(id integer primary key, name text)")
db.execute("insert into users(name) values (?)", "Hamada")
db.query("select count(*) from users") # => ((1))

liteb provides tight Rails/ActiveRecord integration and can be configured as follows

In the database.yml

adapter: litedb
# normal sqlite3 configuration follows

liteb offers integration with the Sequel database toolkit and can be configured as follows

DB = Sequel.connect("litedb://path_to_db_file")

litecache is a high speed, low overhead caching library that uses SQLite as its backend. The litecache can be accessed from multiple processes on the same machine seamlessly. It also has features like key expiry, LRU based eviction and increment/decrement of integer values.

require 'litestack'
cache = Litecache.new(path: "path_to_file")
cache.set("key", "value")
cache.get("key") #=> "value"

In your desired environment file (eg production.rb)

config.cache_store = :litecache, {path: './path/to/your/cache/file'}

It provides a transparent integration that uses the Rails caching interface

litecache spawns a background thread for cleanup purposes. In case it detects that the current environment has Fiber:: Scheduler or Polyphony load it will generate a thread instead, to save both memory and CPU cycles.

More information about Litejob can be found at litejob guide

litejob is a fast and efficient job queue processor for Ruby applications. It also builds on top of SQLite, which provides transaction guarantees, persistence and exceptional performance.

require 'litestack'
# define your job class
class MyJob
  include ::Litejob
      
  queue = :default
      
  # must implement perform, with any number of params
  def perform(params)
    # do stuff
  end
end
    
#schedule a job asynchronusly
MyJob.perform_async(params)
    
#schedule a job at a certain time
MyJob.perform_at(time, params)
    
#schedule a job after a certain delay
MyJob.perform_after(delay, params)

In your desired environment file (eg production.rb)

config.active_job.queue_adapter = :litejob

You can add additional configuration to litejob.yml (or config/litejob.yml if you are integrating with Rails)

queues:
    - (default, 1)
    - (urgent, 5)
    - (critical, 10, "spawn")

Queues must include a name and a priority (a number between 1 and 10) and can optionally add the “spawn” token, which means that each job will run in its own context of concurrency (thread or fiber)

This is a drop in replacement adapter for the actioncable replacement async and other production adapters (eg PostgreSQL, Redis). This adapter is currently only tested in local (inline) mode.

Getting up and running with litecable requires configuring your cable.yaml file under the config/ directory

cable.yaml

development:
  adapter: litecable

test:
  adapter: test

staging:
  adapter: litecable

production:
  adapter: litecable

Litesearch adds full text search capabilities to Litedb, you can use it in standalone mode as follows:

require 'litestack/litedb'
db = Litedb.new(":memory:")
# create the index
idx = db.search_index('index_name') do |schema|
    schema.fields (:sender, :receiver, :body)
    schema.field :subject, weight: 10
    schema.tokenizer :trigram
end
# add documents
idx.add({sender: 'Kamal', receiver: 'Laila', subject: 'Are the girls awake?', body: 'I got them the new phones they asked for, are they awake?'})
# search the index, all fields
idx.search('kamal')
# search the index, specific field, partial workd (trigram)
idx.search('subject: awa') 

Litesearch is tightly integrated with ActiveRecord and Sequel, here are examples of the integration

class Author < ActiveRecord::Base
    has_many :books
end

class Book < ActiveRecord::Base
    belongs_to :author

    include Litesearch::Model

    litesearch do |schema|
        schema.fields (:title, :description)
        schema.field :author, target: 'authors.name'
        schema.tokenizer :porter
    end
end
# insert records
Author.create(name: 'Adam A. Writer') 
Book.create(title: 'The biggest stunt', author_id: 1, description: 'a description') 
# search the index, the search method integrates with AR's query interface
Book.search('author: writer').limit(1).all

class Author < Sequel::Model
    one_to_many :books
end

class Book < Sequel::Model
    many_to_one :author

    include Litesearch::Model
    litesearch do |schema|
        schema.fields (:title, :description)
        schema.field :author, target: 'authors.name'
        schema.tokenizer :porter
    end
end
# insert records
Author.create(name: 'Adam A. Writer') 
Book.create(title: 'The biggest stunt', author_id: 1, description: 'a description') 
# search the index, the search method integrates with Sequel's query interface
Book.search('author: writer').limit(1).all

Litestack has a module that can collect useful metrics for its different components, for each component, you need to add the following to the respective .yml file (database.yml in the case of Litedb)

    metrics: true # default is false

When you enable metrics, it starts collecting data from different modules and stores it in a database file called metric.db located in the Litesupport.root folder

Litemetric has an API that enables the collection of arbitrary metrics for non-litestack classes. The metrics are in the database but currently Liteboard can only display correct data for Litestack modules, displaying arbitrary metrics for other components will be included later.

Liteboard is a simple web server that provides a web interface for collected metrics, it should be available worldwide, for the type of usage instructions

It allows you to point to a specific metrics database file or a config file and then it will display the data in the metrics database.

Example metric views:

• Database size, number of tables and indexes
• Number of read/write questions
• Read/Write query ratio over time
• Reading/Writing question over time
• Slow questions
• Most expensive queries (total run time = frequency * cost)

• Cache size, % of size limit
• Number of entries
• Reads/Writes over time
• Read the hits/misses over time
• Most written entries
• Most read entries

Bug reports and pull requests are welcome on GitHub at https://github.com/oldmoe/litestack.

The gem is available as open source under the terms of MY License.