thebernardlim

Articles on Cloud, Data and Software Engineering

25 days of Serverless - Challenge 2

Solution to 25 days of Serverless - Challenge 2

For Challenge #2 of the ‘25 days of Serverless’ challenge, we are to implement a solution to send Lucy reminders regarding coffee-making steps and candle-lighting at various times accordingly. I d...

Azure Data Factory - How to connect to Azure File Storage?

Steps to show how to connect to Azure File Storage / Share in Azure Data Factory

When trying to use Azure File Storage as a source in Azure Data Factory, you might wonder where you would get the credentials to connect to Azure File Storage in Data Factory. Here are the steps: ...

Azure Job Scheduling Options for Azure SQL DB

Article showing/comparing a list of Azure services we can use for scheduling jobs required for Azure SQL database

We recently were faced with a request to run SQL scripts (stored procedures) for an Azure SQL Database on schedule. Naturally, SQL Job Agents came into my mind. However, Azure SQL Single Databases...

Azure Data Factory - Removing Triggers without Errors

Removing triggers from ADF pipelines without errors

In the ‘Author’ page, if you attempt to remove triggers by clicking on ‘Trigger’ button, there might be a change you would attempt to click on ‘X’ next to the trigger you want to delete, i.e. trigg...

Azure Elastic Jobs - Setup Guide

Guide on setting up / creating Azure Elastic Jobs

Azure Elastic Jobs is a feature for Azure SQL DB which allows for scheduled jobs (For database automation or maintenance activities) to be run. It involves a number of components, namely: Crea...

Azure Data Engineer Associate (DP-200, DP-201) Certification Study Guide

Sharing my experience to getting certified for the Azure Data Engineer Associate certification

I had just recently cleared both DP-200 and DP-201 exams for the Azure Data Engineer Associate certification. Below are the list of steps I undertook to achieve them. Hope you will find them useful...

Azure Streaming Analytics - Export / Import Streaming Analytics Jobs

Export / Import Streaming Analytics using Visual Studio Code

Unlike other services, there is no ‘Export Template’ option within the portal as shown in an example below: Exporting One way to export a Streaming Analytics job is through Visual Studio...

Azure Data Engineer Associate Certification Study Notes

Azure Data Engineer Associate certification study notes

Azure Data Engineer Associate Certification Study Notes Azure Databricks Cluster Modes Standard Terminate after 120 seconds Python, R, Scala, SQL For single users ...

Azure IOT Hubs vs Event Hubs - What are the differences?

Quick summary on the differences between IoT Hubs vs Event Hubs

Both Event Hub and IOT Hub are used for stream ingestion. So when do we use which? Below is a quick summary: Difference #1 : Communication Event Hub: 1 way communication from devices / sources ...

Azure Data Factory - How to inspect schema of Avro files?

ADF is unable to infer schema of Avro files. Here is the solution.

Problem Avro file format is the only format when capturing events from Event Hub. However Avro files cannot be processed directly (To inspect schema) on Data Factory. Solution Use a Databricks ...