Short Message Compression using Large Language Models

• Example of compression:

  ./ts_sms c "Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression."
  뮭䅰㼦覞㻪紹陠聚牊
    

• Example of decompression:

  ./ts_sms d 뮭䅰㼦覞㻪紹陠聚牊
  Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression.
  

• Comparison with brotli which embeds a dictionary to optimize the compression of small messages:

  ./ts_sms c "Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression." -F base64
  9mY1wFk1VUFdUjvxC0wQfA==
  

  echo -n "Lossless compression reduces bits by identifying and eliminating statistical redundancy. No information is lost in lossless compression." | brotli | base64
  H4cAYEVPlqqQnpIJ3RBsRngCqhywb5ksOXr9JgfAu42tLx5BpYJuPXKKY9U0Rn1dxoVKWgFFY8Bz
  0zceYpcspm/ft+EqAA==
  

Downloads

Technical information

ts_sms works similar to ts_zip. It uses a specific padding system compatible with arithmetic coding so that the length of the message does not need to be explicitly encoded.

Fabrice Bellard – https://bellard.org/