With the incredible advancements in the cloud, many companies are looking to migrate their workloads in order to save costs and stay relevant in these fast-changing times. A large part of the migration relates to moving data workloads. While all cloud providers provide some sort of managed relational database service, each provider has distinct advantages. There are numerous links and blog posts out there to help organizations choose the best cloud provider for their desired workloads, but many times, an important factor is left out of the decision-making process: scheduled jobs and maintenance tasks.
For this article, let us focus on AWS and Microsoft Azure. They each have strong points when it comes to moving data workloads to the cloud, especially when it comes to moving the applications supported by that data. Let’s take a look at each provider and compare some strengths and weaknesses when it comes to scheduled jobs.
Microsoft Azure
Microsoft Azure is a common choice for organizations that primarily run on the Microsoft stack. It makes logical sense to move your existing workloads to the provider that creates your current application stack, which should make the migration easier, and you are still running on supported architectures, since Microsoft is the creator of the codebase, so it should easily migrate to the cloud.
When it comes to databases, Microsoft Azure supports Azure SQL Database, which is touted as being as close as possible to the same as users are used to in an on-prem environment. This is largely true, as SQL Server Management Studio (SSMS) can still be used to access it, and almost all functionality still functions exactly as you would expect it to while running in the cloud. One of the largest differences between SQL Server and Azure SQL Database is the lack of the SQL Server Agent. Many organizations use the SQL Server Agent to schedule maintenance tasks like integrity checks, backups, and other business logic tasks. While Azure, like other cloud providers, handles the backups automatically as part of the service, it does not automate all the other maintenance tasks that organizations use to maintain their data structures.
Azure does provide the Azure Data Factory, which can be used to create integrations and processes similar to SSIS (SQL Server Integration Services), and they can be configured to run on a schedule, it is a different developer experience and different UI that many developers of maintenance tasks may need some time to get used to. While there are methods to migrate existing SSIS packages to Azure Data Factory, each one of these would need to be re-tested, and possibly modified, to work seamlessly with Azure SQL Database.
If existing jobs can be migrated, why should I be concerned? With relatively new technologies (such as cloud services), the new service may not include all the functionality from the on-prem service, until the provider gets enough feedback to make it worthwhile, to build out that functionality. This means that some packages may not function after a direct migration to Azure Data Factory. In addition, it is possible to have scheduled tasks that do not rely solely on SSIS packages to complete. These may also not be easily covered by Azure Data Factory. While some features may be replicated automatically by the service, and some may be available with other cloud-provider services, it may not be a simple task to ‘lift-and-shift’ existing scheduled jobs and maintenance tasks to cloud providers.
Amazon Web Services (AWS)
AWS, on the other hand, provides SQL Server as a possible engine for its RDS (Relational Database Service) service. This also allows direct client connections and management from SSMS, making it appear nearly identical to end users, which eases the cloud-adoption headaches that learning new technologies may bring. AWS RDS SQL Server also includes the SQL Server Agent directly, making job development the same, and managing and configuring jobs nearly identically to existing methods. This is particularly interesting because AWS provides the SQL Agent, yet Azure, a Microsoft product, does not. This may be one of the reasons why AWS currently provides more Microsoft workloads than Azure does. AWS also provides a much larger set of features (at the time of this writing), making it more versatile for upgraded, cloud-native apps and other new business applications.
While AWS also has other features, such as Simple Workflow Service, which may be used to create similar tasks, the migration may be simpler and not require picking up new skill sets immediately.
While there are many more factors in choosing cloud providers than just scheduled jobs and maintenance tasks, these tasks do need to be considered before making the decision. The other major factor, also unrelated to jobs, but important to the provider choice, includes future business plans. It may make sense to move to Azure since you are currently using the Microsoft stack. But will that use continue, or will many more organizations move to microservices and develop completely new apps, which may not rely solely on the Microsoft stack? Each organization must make its own decision on this, and they will need to consider these, as well as many other factors, in making that decision.
Written by Andy Maser, Data Architect
