Published in ITNEXT·5 hours agoTrino Gateway reloaded in Chango Cloud in PreviewTrino gateway is a proxy which routes trino queries to upstream backend trino clusters dynamically. As I mentioned in the previous posts, Route trino queries dynamically using Trino Gateway and Trino Gateway , if one of the backend trino clusters has been exhausted, trino gateway will route the queries to…Trino4 min readTrino4 min read
Published in ITNEXT·Dec 10, 2022Simple Streaming Application to ingest Json to Iceberg Table in S3If you want to build streaming applications to ingest events to storage like S3, you may consider event streaming platform like kafka, and streaming applications like spark streaming application to consume events from kafka topics, and transform and save them to S3. But with some reasons like management issues, if…Iceberg8 min readIceberg8 min read
Published in ITNEXT·Oct 14, 2022The way to integrate Trino ETL Jobs using dbt-trino with Airflow on KubernetesTrino is the most popular query engine in data lakehouses. Recently, trino can be used to run long running ETL jobs with fault tolerant execution configuration as well as interactive queries, which means, I think, you can replace Hive with trino for most of the cases. Such trino etl jobs…Dbt6 min readDbt6 min read
Published in ITNEXT·Oct 9, 2022Things to consider to submit Spark Jobs on Kubernetes in cluster modeIt is hard to submit spark jobs on kubernetes. As mentioned in the previous post of Hive on Spark in Kubernetes in which it is shown that spark thrift server as a usual spark job submitted to kubernetes, there are many things to consider to submit spark jobs onto kubernetes. …Spark9 min readSpark9 min read
Published in ITNEXT·Jul 14, 2022Trino GatewayPreviously, I have mentioned how to build a trino gateway to route queries dynamically to downstream trino clusters in this article. Here, I am going to show the improved trino gateway architecture using DataRoaster Trino Controller to control all the trino ecosystem components to create a nice trino gateway easily. Trino Gateway ArchitecturePresto5 min readPresto5 min read
Published in ITNEXT·Jun 15, 2022Route trino queries dynamically using Trino GatewayTrino is a popular query engine to query data in data lakehouses. Let’s say there is just one big trino cluster consisting of lots of nodes, but such a big trino cluster can be problematic for some organizations. Trino can be used for ETL workloads and interactive query. …Kubernetes8 min readKubernetes8 min read
Published in ITNEXT·Dec 12, 2021Hive on Spark with Spark OperatorSpark Thrift Server is used as Hive Server whose execution engine is spark. As mentioned in Hive on Spark in Kubernetes, Spark Thrift Server can be deployed onto Kubernetes. For this case,spark-submit installed on local machine has been used to submit spark thrift server to kubernetes. There is another way…Kubernetes9 min readKubernetes9 min read
Published in ITNEXT·Sep 9, 2021DataRoaster is now open-sourced, why I created itDataRoaster is a tool to provide data platforms running on kubernetes. Recently I have open-sourced it. Before I developed DataRoaster, I used free data platforms like HDP(Hortonworks Data Platform) to build data lakes. After Hortonworks was acquired by Cloudera, HDP was not free any more. To build a data lake…Hive2 min readHive2 min read
Published in ITNEXT·May 21, 2021Trino on NomadTrino(formerly PrestoSQL) is a popular distributed interactive query engine in data lake. Trino can be used as not only query engine, but also data preparation engine in data lake. As data platform component, Trino is one of my favorite components to use in data lake. …Presto9 min readPresto9 min read
Published in ITNEXT·May 20, 2021Elasticsearch on NomadAs I mentioned in the previous post, I have been looking for an alternative to Kubernetes to deploy stateful applications on container orchestrators. Generally, stateful applications need volumes to persist data. The volumes should be provisioned dynamically when the stateful application job is submitted. But currently, Nomad does not support…Nomad12 min readNomad12 min read
