Eric Sun

Published in The Startup

·Pinned

Data Dependency Driven Orchestration

AirFlow and Prefect are probably the most popular schedulers in 2021. They are both more data-aware than the traditional orchestration softwares. This article will describe an additional service architecture to put the data dependency as the enabling pattern for an effective & efficient orchestration in the complex big data environment…

Orchestration

10 min read

Published in Analytics Vidhya

·Pinned

Are We Taking Only Half Of The Advantage Of Columnar File Format?

(* originally posted in LinkedIn in 2018 ) Columnar file formats have become the primary storage choice for big data systems, but when I Googled related topics this weekend, I just found that most articles were talking about the simple query benchmark and storage footprint comparisons between a particular columnar…

Columnar

8 min read

Published in The Startup

·Nov 22, 2020

Lego vs SoC, Apple M1 + MT8195, Microservices and Big Data Model

This week (2020–11–10) was really big for System on a Chip: first Apple M1, and then followed by MediaTek MT8195/MT8192. But why on earth these have anything to do with lego, microservices and even data model? It is a topic I have put off for a few years. Probably, all…

Soc

6 min read

Lego vs SoC, Apple M1 + MT8195, Microservices and Big Data Model

Published in The Startup

·Mar 16, 2020

3.5 Musketeers to Reshape Data Lake

“Data Lake” has become a buzz word since 2016, and we even invented the phrase “Lakehouse” lately. Did we simply upload the web logs and dump tables from MPP database into HDFS or cloud storage, so we call that Data Lake? The recent rise of Data Lake is actually a…

Delta Lake

8 min read

Reshape Data Lake: Delta, Iceberg, Hudi, or Hive

Advocate best practice of big data technologies. Challenge the conventional wisdom. Peel off the flashy promise in architecture and scalability.

Following

See all (9)

Help
Status
Writers
Blog
Careers
Privacy
Terms
About
Knowable

Data Dependency Driven Orchestration

Are We Taking Only Half Of The Advantage Of Columnar File Format?

Lego vs SoC, Apple M1 + MT8195, Microservices and Big Data Model

3.5 Musketeers to Reshape Data Lake

Eric Sun

Coinbase

Netflix Technology Blog

Cassie Kozyrkov

Manish Jain

Jeff Chen