Published inAWS in Plain English·PinnedMWAA: Apache Airflow on AWSStop Worrying, Start DAG’ing — Airflow is an amazing platform to programmatically author, schedule, and monitor workflows. Check out MWAA: Apache Airflow on AWS Part 1 to get an overview of what Airflow is and how we can use it to automate our data pipelines and workflows. Nothing makes sense until you start implementing, so…AWS6 min readAWS6 min read
Published inSelectFrom·PinnedApache Spark: All about SerializationOverview of How You Can Tune Your Spark Jobs to Improve Performance In distributed systems, data transfer over the network is the most common task. If this is not handled efficiently, you may end up facing numerous problems, like high memory usage, network bottlenecks, and performance issues. Serialization plays an…Apache Spark3 min readApache Spark3 min read
PinnedIntroduction to Akka Typed with Scala for beginners.Learn to Build distributed and fault tolerant systems with me and Dead-pool. Learn best in class technology upon which many cutting edge frameworks are build(Play Framework, lagom,cloudFlow, etc…) the list goes on… I would like to share my insights on how to get there.when i started my journey to learn…Scala8 min readScala8 min read
Sep 8Kaggle-Databricks-Snowflake: End-to-End Hands-on ETL demoEffortless Data Integration: Kaggle, Databricks, and Snowflake in Action There has been a notable shift in focus, primarily towards the collection and storage of data, due to advancements in artificial intelligence and machine learning. The goal is to gather as many data points as possible. But it’s crucial to understand…Data6 min readData6 min read
Published inITNEXT·Sep 7Security at Scale: Sensitive Data Supervision with Regulatory ConfidenceData — a dynamic entity that orchestrates our current course — navigates us through our present and opens up endless future possibilities. It is at the epicenter of all our endeavors, making data security essential for reliability and trustworthiness. …Data5 min readData5 min read
Published inITNEXT·Aug 7Mitigating CVE Shock, AKA Navigating the Perils of Vulnerability HellDevOps teams customarily abide by a typical pattern concerning overall security and vulnerabilities: they overburden themselves with unwanted vulnerability intricacies. Commonly referred to as “CVE shock”, this is a phase in which DevOps teams get distracted and feel helpless when exposed to an overwhelming list of vulnerabilities to mitigate and…DevOps5 min readDevOps5 min read
Jun 21Building a custom WhatsApp AI chatbot with ChatGPT and Selenium.Deep dive into ChatAgent development and Prompt Engineering. Introduction: Today we are going to see how we can integrate and automate chatCPT on WhatsApp and convert it into an AI chatbot. It sometimes feels overwhelming to connect to OpenAI or dall-E to generate responses or images right. I felt it too…ChatGPT2 min readChatGPT2 min read
Apr 30Dynamic data load to AWS S3 with Snowpark.The data war between Databricks and Snowflake are inevitable and are rushing the platforms to adapt and onboard new features at an alarming pace. They both have their pros and cons. The advantages are appealing only based on your role and the tasks you want to complete from the platform. …Snowflake4 min readSnowflake4 min read
Published inAWS in Plain English·Apr 5Develop and Invoke AWS Lambda Functions programmatically.Introduction: AWS Lambda is a serverless compute service that allows you to run code without provisioning or managing servers and you can pay only for the compute when the lambda is invoked. This sounds convenient, but the pricing differs when we opt for general-purpose and on-demand computing. When you have a…AWS4 min readAWS4 min read
Published inDataTrek·Feb 9Mastering Tools and Libraries are Trivial.Choose wisely, learn mindfully, implement efficiently, and grow immensely. Today, everybody wants to solve a problem, be it technical or otherwise, and tools and libraries are emerging at an alarming pace like never before. We live in an era where there is no room for the word impossible. …Programming3 min readProgramming3 min read
