Warp Solutions

ClickHouse is an open-source column-oriented DBMS optimized for online analytical processing in real time using SQL. It is a good match for WarpStreams Kafka-compatible clusters. In this Solution, Shawn will set up a ClickHouse account, connect it to WarpStream using their built-in connection wizard, consume over 4 million records, and perform some basic queries. WarpStream - www.warpstream.com ClickHouse - www.clickhouse.com Docs - https://docs.warpstream.com/warpstream/reference/integrations/clickhouse #apachekafka #clickhouse #datastreaming #dataengineering

Series: Warp Solutions Subject: Use Bento and WarpStream to Parquet and query with DuckDB Apache Parquet is an open-source, column-oriented data file format designed for efficient data storage and retrieval. It forms the backbone of many datalake and table format systems. In this Solution, Shawn will create a small pipeline script with the popular open-source Bento tool, to read from a topic in a WarpStream cluster, and write batches of Parquet files which are then queried with #DuckDB. WarpStream - www.warpstream.com Parquet - parquet.apache.org Bento docs - https://warpstreamlabs.github.io/bento/docs/guides/getting_started/ #apachekafka #apacheiceberg #parquet #datastreaming #dataengineering #duckdb #bento

Series: Warp Solutions Subject: Use Bento and WarpStream to populate and use SQLite SQLite is an open-source, embedded, serverless RDBMS that is popular for its small size and ease of use. In this Solution, Shawn will create a small pipeline script with the popular open-source Bento, too, to read from a topic in a WarpStream cluster into a table in SQLite. WarpStream - www.warpstream.com SQLite - www.sqlite.org Docs - https://docs.warpstream.com/warpstream/reference/integrations/sqlite Bento docs - https://warpstreamlabs.github.io/bento/docs/components/buffers/sqlite/#examples Integration docs - https://docs.warpstream.com/warpstream/reference/integrations/parquet #apachekafka #warpstream #sqlite #datastreaming #dataengineering

Series: Warp Solutions Subject: Learn how to set up InfluxDB to consume from WarpStream InfluxDB is an open-source time series database that is a perfect companion to WarpStream's Apache Kafka compatible clusters. Shawn will take you through the steps involved in setting up InfluxDB to consume from topics in WarpStream. WarpStream - www.warpstream.com InfluxDB - www.influxdata.com Integration documentation - https://docs.warpstream.com/warpstream/reference/integrations/influxdb #apachekafka #warpstream #influxdb #datastreaming #dataengineering

Series: Warp Solutions Subject: Use Bento and WarpStream to populate and use ParadeDB ParadeDB is an open-source Elasticsearch alternative built on Postgres. It aims to improve native full-text search capabilities in Postgres. Shawn will walk through using Bento to create a pipeline to consume from a WarpStream topic, write to ParadeDB, generate the indexes, and run through some sample queries. WarpStream - www.warpstream.com ParadeDB - docs.paradedb.com/introduction Bento - github.com/warpstreamlabs/bento Integration docs - docs.warpstream.com/warpstream/reference/integrations/paradedb #apachekafka #warpstream #postgres #paradedb #datastreaming #dataengineering

Series: Warp Solutions Subject: Use Streamlit to dynamically browse WarpStream topics Shawn writes a small Python program that utilizes the Streamlit and Kafka Python libraries to build a simple web form that allows you to browse and consume records from arbitrary clusters and topics dynamically. WarpStream - www.warpstream.com Streamlit - streamlit.io Project docs - https://docs.warpstream.com/warpstream/reference/integrations/streamlit #apachekafka #warpstream #snowflake #streamlit #datastreaming #dataengineering

Series: Warp Solutions Subject: Use DuckDB extension Kwack to read directly from WarpStream Shawn uses the new DuckDB extension, Kwack, to connect to two Apache Kafka topics in WarpStream, read them, and join them. After that, he illustrates how to export the topics to a Parquet file or DuckDB database. WarpStream - www.warpstream.com DuckDB - duckdb.org Kwack - github.com/rayokota/kwack (requires Java 11 or later) ShadowTraffic was used for the creation of the sample data, available at shadowtraffic.io Harlequin was used as a terminal-based SQL IDE, available at https://harlequin.sh/ #apachekafka #warpstream #streambased #duckdb #datastreaming #dataengineering

Series: Warp Solutions Subject: Use Estuary to pipe data from WarpStream to MotherDuck Guest: Dani Pálma - Data Engineer at Estuary Dani and Shawn spin up a serverless WarpStream Apache Kafka-compatible cluster and then use ShadowTraffic to pipe simulated data for Customers and Orders into it. After that, they configure Estuary to connect to the WarpStream cluster as a source, and MotherDuck as a destination. Finally, the data is piped into MotherDuck and basic queries are performed. www.warpstream.com www.estuary.dev ShadowTraffic was used for the creation of the sample data, available at shadowtraffic.io MotherDuck was used as the destination for the streaming data. #apachekafka #warpstream #streambased #duckdb #datastreaming #dataengineering

Series: Warp Solutions Subject: Simulating data for Apache Kafka Guest: Michael Drogalis - Founder of ShadowTraffic Michael and Shawn discuss how ShadowTraffic can create realistic test data for most use cases, even maintaining key values between multiple data simulations. They then cover the technology and load data into WarpStream using ShadowTraffic. www.warpstream.com www.shadowtraffic.io ShadowTraffic docs for Kafka: https://docs.shadowtraffic.io/connections/kafka/ #apachekafka #warpstream #streambased #sql #datastreaming #dataengineering

Series: Warp Solutions Subject: Query streaming data with Arroyo from WarpStream producers Guest: Micah Wylde - CEO of Arroyo Micah and Shawn set up a WarpStream serverless Kafka-compatible cluster, generated some credentials, and then produce data to it using Shadowtraffic. After that, Arroyo came in to read the streams and create pipelines with the data. www.warpstream.com www.arroyo.systems Shadowtraffic was used for the creation of the sample data, available at shadowtraffic.io #apachekafka #warpstream #streambased #sql #datastreaming #dataengineering