How do you or your analytics team handle errors in a pipeline? By pipeline, I mean any type of low-code, no-code, all-code solution that runs on a schedule and performs some sort of activity. Because here’s the truth. Pipelines fail. It doesn’t matter if it’s a ETL, ELT, or application integration pipeline. It will fail. It doesn’t matter if you have a Dev, Test, Prod environment. It will fail. It doesn’t matter if you have 1 person or 10 people supporting it. It will fail. Eventually. So when it does fail (even if it’s the 0.01% of the time). Do you know? The simplest way to know is by setting up error notifications and success notifications via email. Another solution is to have a log file or table in a database. No matter how you do it, it is important to know when your pipeline fails. All the Best, Tucker p.s. Yes, I am writing this because I had a pipeline fail that I did not know about. |