Overview of How You Can Tune Your Spark Jobs to Improve Performance In distributed systems, data transfer over the network is the most common task. If this is not handled efficiently, you may end up facing numerous problems, like high memory usage, network bottlenecks, and performance issues. Serialization plays an…

Learn to Build distributed and fault tolerant systems with me and Dead-pool. Learn best in class technology upon which many cutting edge frameworks are build(Play Framework, lagom,cloudFlow, etc…) the list goes on… I would like to share my insights on how to get there.when i started my journey to learn…

An efficient way to make your spark job flexible. Apache spark offers a magnitude of options to handle and deal with big data at scale, making it one of the most sought-after skills in big data and data engineering. Almost all of the tutorials online show us how to define…

Efficiently transform data in the cloud data warehouse. — Today data is getting captured from almost everything that can produce 0/1. This data can put a dent in your operational costs if the business intends only to store it in a database. …

In our cloud-agnostic digital world, security is a crucial aspect, be it a Fortune 500 company or a small service provider gaining traction. The importance of security elevates when deployment relies on cloud-based environments. And all companies rely extensively on microservices when deploying an application or integrating a feature onto…

How to read data from S3 using boto3 and python, transform using Scala. Designing and developing data pipelines is at the core of big data engineering. ETL is a major job that plays a key role in data movement from source to destination. Extracting data from Sources can be daunting…

Speed over perfection is very crucial when a new service is being released to the market. First-comers gain a lot of traction, and there is a high likelihood of early adoption. On the other hand, sometimes, rushing to deliver the product leads to minimal review and testing. This leaves the…

Learn how to convert a nested JSON file into a DataFrame/table Handling Semi-Structured data like JSON can be challenging sometimes, especially when dealing with web responses where we get HTTP responses in JSON format or when a client decides to transfer the data in JSON format to achieve optimal performance…

Catch up on the trending articles from Data Engineering Space. A curated list of the most engaging blog posts will be shared here and a newsletter will be sent out to keep the readers up-to-date with the world of Data Engineering. Data is the epicenter of the Digital world. Every…