All posts by Sander Rossel

Azure Functions and Serverless Computing

The post “Azure Functions and Serverless Computing” appeared first on MSDN Azure Development Community.

In my previous blog post, WebJobs in Azure with .NET Core 2.1, I briefly mentioned Azure Functions. Azure Functions are usually small (or somewhat larger) bits of code that run in Azure and are triggered by some event. Azure takes complete care of the entire infrastructure of your Functions making it a so-called serverless solution. The only thing you need to worry about is your code.

The code samples for this post can be found on my GitHub profile. You shouldn’t really need it because I’m only using default templates, but I’ve included them because I needed them in source control for CI/CD anyway.

Serverless computing

Before we get into Azure Functions I’d like to explain a bit about serverless computing. Serverless computing isn’t completely serverless, your code still needs somewhere to run after all. However, the servers are completely managed by your cloud provider, which is Azure in our case.

With most App Services, like a Web App or an API App, you need an App Service Plan, which is basically a server that has x CPU cores and y memory. While you can run your Functions on an App Service Plan it’s far more interesting to run them in a so-called “Consumption Plan”. With a consumption plan, your resources are completely dynamic, meaning you’re only actually using a server when your code is running.

It’s cheap

Let me repeat that, you’re only actually using a server when your code is running. This may seem trivial, but has some huge implications! With an App Service Plan, you always have a server even when no code is running. This means you pay a monthly fee just to keep the server up and running. See where this is heading? That’s right, with a consumption plan you pay only when your code is running because that’s the only time you’re actually using resources!

There’s a pretty complicated formula using running time, memory usage and executions, but believe me, Functions are cheap. Your first million executions and 400,000 GB/sec (whatever that means) are free. Microsoft has a pricing example which just verifies that it’s cheap.

It’s dynamic

Price is cool, but we ain’t cheap! Probably the coolest feature about Functions is that Azure scales servers up and down depending on how busy your function is. So let’s say you’re doing some heavy computing which takes a while to run and is pretty resource intensive, but you need to do that 100 times in parallel… Azure just spins up some extra servers dynamically (and, of course, you pay x times more, but that’s still pretty cheap). When your calculations are complete your servers are released and you stop paying. There is a limit to this behavior. Azure will spin up a maximum of 200 server instances per Function App and a new instance will only be allocated at most once every 10 seconds. One instance can still handle multiple executions though, so 200 servers usually does not mean 200 concurrent executions.

That said, you can’t really configure a dynamically created server. You’ll have to trust Microsoft that they’re going to temporarily give you a server that meets your needs. And, as you can guess, those needs better be limited. Basically, your need should be that your language environment, such as the .NET Framework (or .NET Core) for C#, is present. One cool thing though, Next to C#, Functions can be written in F# and JavaScript, and Java is coming in v2. There are some other languages that are currently running in v2, like Python, PHP, and TypeScript, but it looks like Microsoft isn’t planning on fully supporting these languages in the near future.

Azure Functions in the Azure Portal

Let’s create our first Azure Functions. Go to App Services and create a new one. When you create an App Service you should get an option to create a Function App (next to Web App, API App, etc.). The creation screen looks pretty much the same as for a regular web app, except that Function Apps have a Consumption Plan as a Hosting Plan (which you should take) and you get the option to create a Storage Account, which is needed for Azure to store your Functions.

Create a new Function App
Create a new Function App

This will create a new Function App, a new Hosting Plan, and a new Storage Account. Once Azure created your resources look up your Function Apps in your Azure resources. You should see your version of “myfunctionsblog” (which should be a unique name). Now, hover over “Functions” and click the big plus to add a Function.

Azure Functions
Azure Functions

In the next form, you can pick a scenario. Our options are “Webhook + API”, “Timer” or “Data processing”. You can also pick a language, “CSharp”, “JavaScript”, “FSharp” or “Java”. Leave the defaults, which are “Webhook + API” and “CSharp” and click the “Create this function” button.

Editing and running your Azure Functions

You now get your function, which is a default template. You can simply run it directly in the Azure portal and you can see a log window, errors and warnings, a console, as well as the request and response bodies. It’s like a very lightweight IDE in Azure right there. I’m not going over the actual code because it’s pretty basic stuff. You can play around with it if you like.

Next to the “Save” and “Run” buttons you see a “</> Get function URL” link. Click it to get the URL to your current function. There are one or more function keys, which give access to just this function); there are one or more host keys, which give access to all functions in the current host; and there’s a master key, which you should never share with third parties. You can manage your keys from the “Manage” item in the menu on the left (with your Functions, Proxies, and Slots).

Consuming Azure Functions

The Function we created is HTTP triggered, meaning we must do an HTTP call ourselves. You can, of course, use a client such as Postman or SoapUI, but let’s look at some C# code to consume our Function. Create a (.NET Core) Console App in Visual Studio (or use my example from GitHub) and paste the following code in Program.cs.

If your Function returns an error (for whatever reason) your best logging tool is Application Insights, but that is not in the scope of this blog post.

Of course, if you have a function that’s triggered by a timer or a queue or something else you don’t need to trigger it manually.

Azure Functions in Visual Studio 2017

Creating a Function in the Azure Portal is cool, but we want an IDE, source control, and CI/CD.  So open up VS2017 and look for the “Azure Functions” template (under Visual C# -> Cloud). You can choose between “Azure Functions v1 (.NET Framework)” and “Azure Functions v2 Preview (.NET Standard)”. We like .NET Standard (which is the common denominator between the .NET Framework, .NET Core, UWP, Mono, and Xamarin) so we pick the v2 Preview.

Again, go with the HTTP trigger and find your Storage Account. Leave the access rights on “Function”. The generated function template is a little bit different than the one generated in the Azure portal, but the result is the same. One thing to note here is that Functions actually use the Microsoft.Azure.WebJobs namespace for triggers, which once again shows the two can (sometimes) be interchanged.

When you run the template you get a console which shows the Azure Functions logo in some nice ASCII art as well as some startup logging. Windows Defender might ask you to allow access for Functions to run, which you should grant.

Now if you run the Console App we created in the previous section and change the functions URL to “http://localhost:7071/api/Function1” (port may vary) you should get the same result as before.

Deployment using Visual Studio

If you’ve read my previous blog posts you should be pretty familiar with this step. Right-click on your Functions project and select “Publish…” from the menu. Select “Select Existing” and enable “Run from ZIP (recommended)”. Deploying from ZIP will put your Function in a read-only state, but it most closely matches your release using CI/CD. Besides, all changes should be made in VS2017 so they’re in source control. In the next form find your Azure App and deploy to Azure.

If you’ve selected Azure Functions v2 earlier you’ll probably get a dialog telling you to update your Functions SDK on Azure. A little warning here, this could mess up already existing functions (in fact, it simply deleted my previous function). As far as I understand this only affects your current Function App, but still use at your own risk.

Once again, find your Functions URL, paste it in the Console App, and check if you get the desired output.

Deployment using VSTS

Now for the good parts, Continuous Integration and Deployment. Open up Visual Studio Team Services and create a new build pipeline. You can pick the default .NET Core template for your pipeline, but you should change the “Publish” task so it publishes “**/FunctionApp.csproj” and not “Publish Web Projects”. You can optionally enable continuous integration in the “Triggers” tab. Once you’re done you can save and queue a build.

The next step is to create a new release pipeline. Select the Azure “App Service deployment” template. Change the name of your pipeline to something obvious, like “Function App CD”. You must now first add your artifact and you may want to enable the continuous deployment trigger. You can also rename your environment to “Dev” or something.

Next, you need to fill out some parameters in your environment tasks. The first thing you need to set is your Azure subscription and authorize it. For “App type” pick the “Function App”. If you’ve successfully authorized your subscription you can select your App Service name from the drop-down. Also, and this one is a little tricky, in your “Azure App Service Deploy” task, disable “Take App Offline” which is hidden under the “Additional Deployment Options”. Once you’re done, and the build is finished, create a new release.

When all goes well, test your changes with the Console App.

If everything works as expected try changing the output of your Function App, push it to source control, and see your build and release pipelines do their jobs. Once again, test the change with your Console App.

Working around some bugs

So… A header that mentions bugs and workarounds is never a good thing. For some reason, my release kept failing due to an “invalid access to memory location”. Probably because I had already deployed the app using VS2017. I also couldn’t delete my function because the trash can icon was disabled. Google revealed I wasn’t the only one with that problem. So anyway, I am currently using a preview of Azure Functions v2 and I’m sure Microsoft will figure this stuff out before it goes out of preview.

Here’s the deal, you probably have to delete your Function App completely (you can still delete it through your App Services). Recreate it, go to your Function App (not the function itself, but the app hosting it, also see the next section on “Additional settings”), and find the “Function app settings”. Over here you can find a switch “~1” and “beta” (which are v1 and v2 respectively). Set it on “beta” here. Now deploy using VSTS. Publishing from VSTS will cause your release to fail again.

Bottom line: don’t use VS2017 to deploy your Function App!

Additional settings

There’s just one more thing I’d like to point out to you. While Azure Functions look different from regular Web Apps and Web APIs they’re still App Services with a Hosting Plan. If you click on your Function App you land on an “Overview” page. From here you can go to your Function app settings (which includes the keys) and your Application settings (which look a lot like Web App settings). You’ll find your Application settings, like “APPINSIGHTS_INSTRUMENTATIONKEY”, “AzureWebJobsDashboard”, “AzureWebJobsStorage” and “FUNCTIONS_EXTENSIONS_VERSION”.

Another tab is the “Platform features” tab, which has properties, settings and code deployment options (see my post Azure Deployment using Visual Studio Team Services (VSTS) and .NET Core for more information on deployment options).

Functions Platform features
Functions Platform features

Wrap up

Azure Functions are pretty cool and I can’t wait for v2 to get out of preview and fully support .NET Standard as well as fix the bugs I mentioned. Now, while Functions may solve some issues, like dynamic scaling, it may introduce some problems as well.

It is possible to create a complete web app using only Azure Functions. Whether you’d want that is another question. Maybe you’ve heard of micro-services. Well, with Functions, think nano-services. A nano-service is often seen as an anti-pattern where the overhead of maintaining a piece of code outweighs the code’s utility. Still, when used wisely, and what’s wise is up to you, Functions can be a powerful, serverless, asset to your toolbox. If you want to know more about the concepts of serverless computing I recommend a blog post by my good friend Sander Knape, who wrote about the AWS equivalent, AWS Lambda, The hidden challenges of Serverless: from VM to function.

Don’t forget to delete your resources if you’re not using them anymore or your credit card will be charged.

Happy coding!

WebJobs in Azure with .NET Core 2.1

The post “WebJobs in Azure with .NET Core 2.1” appeared first on MSDN Azure Development Community.

WebJobs aren’t new to Azure or .NET. There’s even a default Azure WebJob template in Visual Studio 2017 for the full .NET Framework. However, a similar template for WebJobs in .NET Core is somehow missing from Visual Studio. In this post, I’m using .NET Core 2.1.

Creating a WebJob in .NET Core isn’t hard, but you have to know some tricks, especially if you want to use some .NET Core goodies like logging and DI.

In this post, we’re going to build a WebJob and release it to Azure using Visual Studio, the Azure portal, and VSTS.

You can find the code samples for this post on GitHub.

What are WebJobs

A WebJob is a program running in the background of an App Service. It runs in the same context as your web app at no additional cost. Maybe you need to do some hourly task or do some cleanup task every night at 1 AM. Azure Application Insights uses a WebJob to report your app’s statistics.

WebJobs can be scheduled, like hourly or daily, but they can also be triggered. A trigger could be a file upload or a new message on a queue.

WebJobs vs. Functions

I’ve often found comparisons between WebJobs and Azure Functions. In a way, Functions are the successors to WebJobs. Functions are (usually) small pieces of code that run in Azure and are, just like WebJobs, triggered at a certain event, including an HTTP trigger.

Functions are often a great alternative to WebJobs, but if you already have a web app it could make sense to use a WebJob instead. Especially if you want to share code and/or settings between the WebJob and your web app as they run in the same context, which also makes deployment quite easy.

Creating a Storage Account

Before we continue let’s take care of something first. A WebJob requires an Azure Storage Account.  I’ll quickly walk you through the process of creating one.

In Azure, find “Storage Accounts” and add one. You’ll have to pick a name that’s unique across Azure. Other than that you can leave the defaults. We’re talking about cents per GB, so don’t worry about costs too much.

Once your Storage Account is ready, select it and find your “Access keys”. We’ll need one of the two connection strings later.

Creating a WebJob

As said, there’s a WebJob template for the full .NET Framework. I recommend you check it out. Start by creating an ASP.NET Web Application and then add a new WebJob. If you try to create the WebJob right away you’ll get an error saying that the project needs to be saved first (although it does create the WebJob).

We’re here for the .NET Core version of a WebJob though. So start by creating an ASP.NET Core Web Application and then add a new .NET Core Console App project to your solution.

The first thing we need to do to is install the Microsoft.Azure.WebJobs package from NuGet. We should also install Microsoft.Azure.WebJobs.Extensions. Here’s the catch though, the latest stable versions of these libraries have dependencies on the full .NET Framework so we’re going to need version 3.0.0-beta5 (at the time of this writing), which is fully compatible with .NET Core.

Other NuGet packages we’ll need are Microsoft.Extensions.Options.ConfigurationExtensions (which also gives us the Microsoft.Extensions.Options package, which we also need), Microsoft.Extensions.DependencyInjection and Microsoft.Extensions.Logging.Console. Be sure to install version 2.1.0 of these packages because there seems to be a bug in .NET Core 2.1 that prevents you from using packages with patch versions, like 2.1.1.

Join the Program

The next thing we need to do is change our Program.cs file. If you’ve created a WebJob using the .NET Framework template you can simply copy and paste the Program.cs file that was generated there (although you might want to change the namespace).

Adding Configuration and DI

So I promised you’d get all the .NET Core goodies like logging and DI. By default, a Console App doesn’t have any of that, but you can add it yourself.

Next, create an appsettings.json file and set the “Copy to Output Directory” property to “Copy always”. The appsettings.json file should have two connection strings as mentioned in the Program.cs template file. These are the Storage Account connection strings we created earlier.

The next thing we need is a custom IJobActivator that can be used to inject dependencies into our classes. It needs to be set on the JobHostConfiguration in the Program class.

Adding a Trigger

After that, create a class and name it Functions (just like in the WebJob template). The Functions class will have the actual code for our WebJob.

Of course, we’ll need to add a trigger. This is different than the full .NET Framework. After all, the template uses a static method, which makes DI impossible. Speaking of DI, notice that we’ve also added the Functions class itself to the DI container.

For simplicity, we’ll use a TimerTrigger, which is triggered with a so-called CRON expression. This simply means it’s triggered at a certain minute, hour, day, etc. In this example, it triggers every minute.

We’ll also need to configure timers on the JobHostConfiguration.

Running the example

If you did everything correctly, or if you’re running my code from GitHub, you should now be able to run the Console App. If you break on exceptions or if you’re watching the Output window you may notice a lot of StorageExceptions. Don’t worry about them and ignore them. This is a bug in the WebJobs library and will not affect your program. It may take a minute for your trigger to go off, so have a little patience.

If you head over to your Azure Storage Account you should see two Blob Containers, “azure-jobs-host-output” and “azure-webjobs-hosts”. There’s quite a lot going on here, but you can just ignore it. I’ve found that my WebJob triggers wouldn’t go off for some reason, deleting the Blob Containers usually helped. Apparently, there’s some state stored in there which isn’t always disposed of properly when (re-)adding and removing WebJobs.

Deploying to Azure

The next thing we want to do is deploy our WebJob to Azure. In order for a WebJob to run it needs some executable script that it can use to get going. Many file types are supported, but for us Windows users it makes sense to use an exe, cmd, bat or PowerShell file.

A Console App used to be an exe file, but in .NET Core, it produces a regular DLL file that we need to start manually. So, create a file and name it “run.cmd” and make sure that it’s encoded in UTF-8 no BOM (you can check this using something like Notepad++). It just needs a single line of code, which is “dotnet NetCoreWebJob.WebJob.dll”. This runs your Console App. Make sure you set the “Copy to Output Directory” of the file to “Copy always”.

One last thing, for some reason Azure WebJobs needs all the dependencies of a WebJob, which means all .NET Core packages we used to build it. You can do this by editing the csproj file and adding “<CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>” to the first <PropertyGroup> (underneath “<TargetFramework>”).

Before we can deploy our WebJob we need to deploy our web app. Right-click the ASP.NET project and click “Publish…”. Simply follow the wizard and Visual Studio will deploy your app for you. You can create a new web app or select an existing one. This step isn’t strictly necessary as you can host stand-alone WebJobs, but this should be familiar and it gives you an App Service we can use for our WebJob.

Deploy using Visual Studio

Deploying WebJobs using Visual Studio should be easy as pie. In fact, you probably already know how to do this (don’t do it yet though). Right-click your WebJob project and click “Publish…”. The following wizard looks a whole lot like the publication of a web app, which we just did. You can pick “Select existing” and pick the Azure web app we just created.

Unfortunately, Microsoft messed up this feature in the worst way possible. Visual Studio will deploy the WebJob with the same name as the project, which is “NetCoreWebJob.WebJob”, except the dot is an illegal character in a WebJob name! This messed up my project so bad I had to manually edit it to make my solution working again. Nice one, Microsoft!

So here’s what you do. At the start of the wizard, where you pick either a new or existing App Service, click the arrow next to “Publish immediately” and pick “Create Profile” instead. Now you can first change the name of your WebJob in the settings and deploy after that. Make sure you don’t select “Remove additional files at destination”  or you’ll remove your web app.

Now, browse to the Azure Portal and look up your web app. You’ll find “WebJobs” in the menu. You’ll see your WebJob, but it’s not doing anything. You need to manually run it by selecting it and clicking “Run”. The status should update to “Running”. You can now check out the logs to see that it actually works. You may see an error about the connection strings, but you can ignore those. If you toggle the output you’ll still see a log is written to the console which lets you know it works! If you don’t see a log right away try waiting a minute or two and don’t forget to manually refresh the output.

WebJobs in Azure
WebJobs in Azure

Deploy using the Azure Portal

When you add a new WebJob you’ll need to fill out some options. You can make up some name, set the type to “Triggered” and the triggers to “Manual”. Your alternatives are a “Continuous” WebJob, which just runs and closes (unless you’ve implemented an endless loop in your application); and a “Scheduled” triggered job, which is basically what we have except we’ve implemented the schedule ourselves.

The “File upload” needs a bit of explanation. Here you can upload a zip file that contains your WebJob. So head over to Visual Studio and build your solution. Then go to the output folder of your WebJob project, something like “MyProject\bin[Debug|Release]\netcoreapp2.1”, and put everything in that folder into a zip file. Then select it in the “File upload” in your new WebJob.

Add WebJob
Add WebJob

It will take a few seconds for Azure to create your WebJob so keep refreshing until it pops up. After that, you have to start it manually again and you can check out the logs.

Deploy using VSTS

Ultimately, we want to add our WebJob to our CI/CD pipeline in VSTS. Unfortunately, this functionality doesn’t come out of the box. Luckily, it’s not very difficult either. If you’re not familiar with builds or releases in VSTS check out one of my previous blogs Azure Deployment using Visual Studio Team Services (VSTS) and .NET Core or ARM Templates to Deploy your Resources to Azure.

When you’re in the Azure Portal find the “App Service Editor (Preview)” of your App Service. This lets you browse all the files in your App Service. One thing we notice is that your WebJob is located in “App_Data\jobs\triggered[WebJob name]”. And since your WebJob is really just the output of the WebJob project build it’s simply a matter of copying your WebJob files to App_Data.

WebJob file location
WebJob file location

The build

So go to VSTS and create a new build. Select your repository, your branch, and select “ASP.NET Core” as a template. We only need to change two things here. We need to change the existing “Publish” task and add a new “.NET Core” task to publish our WebJob.

Change the name of the existing publish task to “Publish web app”, untick the “Publish Web Projects” checkbox, and enter the “Path to project(s)”, which is “**/NetCoreWebJob.csproj”. Also, untick the “Zip Published Projects” and “Add project name to publish path” checkboxes as they will eventually mess up our release.

After that create a new .NET Core task, set the “Command” to “publish” and change the name of the task to “Publish web job”. Again, untick “Publish Web Projects” and set the “Path to project(s)”, which is “**/NetCoreWebJob.WebJob.csproj”. And once again, don’t zip the published projects or add the project name to the publish path. The last step here is the “Arguments” field, which can be copy/pasted from the other publish step, except we’re going to add a bit to it: “–configuration $(BuildConfiguration) –output $(build.artifactstagingdirectory)\App_Data\jobs\triggered\WebJobVSTS”.

VSTS WebJob build
VSTS WebJob build

The release

Last, but not least, is the release. Create a new release in VSTS, pick the “Azure App Service deployment” template and fill in the blanks, which is an artifact and your Azure settings in the Environment. Because we’re not zipping our build we just need to change one setting. In the “Deploy Azure App Service” task is a “Package or folder” setting, which has a default of “[…]/*.zip” which obviously isn’t going to work. Instead, use the browser (the button with “…”) and select your drop folder.

Save it, hit new release and pick your latest build. If all goes well you should see your new WebJob in the Azure Portal!

Wrap up

Hopefully, Microsoft will come with a built-in solution for creating, consuming, and deploying WebJobs to Azure in .NET Core soon. Until that time it’s not particularly difficult, it’s just a matter of knowing what to do.

In this post, we’ve seen the TimerTrigger, but there’s a QueueTrigger, a BlobTrigger, and a FileTrigger as well. Experiment, google, and read the official documentation.

Don’t forget to delete your resources.

Happy coding!

ARM Templates to Deploy your Resources to Azure

Hey all,
Last week we talked about Continuous Integration and Deployment to Azure. We set up an App Service and deployed a .NET Core application using Visual Studio Team Services (VSTS), Azure and Visual Studio. In this post we’re going to explore deployment of your Azure resources, or infrastructure, using the Azure Resource Manager and ARM Templates. In this post we’re going to use Azure, VSTS and Visual Studio 2017 again. I recommend you read my previous post if you’re not already familiar with Azure and VSTS.

Sample code for this post can be found on my GitHub profile.

ARM Templates

So what’s this Azure Resource Manager template, or ARM template, exactly? Basically, it’s just some JSON that describes the resources, like app services, databases and storage accounts, you want to have in Azure. Azure can process this template and create or update our resources based on the template. This is also called Infrastructure as Code or (IaC). I’m assuming you know what JSON is, so I’m not explaining that here.

ARM templates are declarative, meaning you describe what you want rather than how you want it. In practice this means you’re not writing code to actually create the resources, but simply state which resources you want to have.

ARM Templates in Azure

So let’s have a look at what this looks like. If you go to a resource in Azure, any resource, you should be able to get the ARM template from the left side menu. I still have my imunique Web App from my previous post, so let’s see what that looks like. Just find the “Automation Script” menu item.

On the left side of the panel that opens you can find the different nodes that make up the ARM template.

ARM template nodes
ARM template nodes

As you can see this (generated) template has three input parameters, no variables, and three actual resources. A parameter is an input for the script, a variable is a computed value. For now we’re mostly interested in the resources. We have the ‘serverfarm’, or the hosting plan; the actual web app, imunique; and the host name binding, which makes your web app available on imunique.azurewebsites.net (Azure created this for us).

Here’s a part of the actual JSON template describing (a part of) the web app.

ARM templates in Azure
ARM templates in Azure

So there’s some good news and some bad news. The good news is Azure can create most of most ARM templates for us. The bad news is that they aren’t always of high or even usable quality. Of course we can tweak them however we like.

Azure Resource Group Projects

Let’s create our own ARM template. Open up Visual Studio 2017 and create a new “Azure Resource Group” project (found under “Cloud”) and name it whatever you like. I’ve named my solution AzureResourceSamples and the different projects have ARM templates for different resources. Again, you can find it on my GitHub profile.

Once you create the project you can pick a pre-defined template from Visual Studio, like a web app. You can also get a template from the Azure Quickstart templates on GitHub. There’s literally hundreds of them ranging from web apps to VM’s to databases. You can pick an empty template as well, it’s possible to add resources later. I’ve picked the “Web app” from the Visual Studio templates. This creates a web app with a hosting plan, Application Insights and auto-scaling. The ARM template can be found in the WebSite.json file. I can recommend studying it thoroughly.

Azure Resource Group project
Azure Resource Group project

The first thing we can do is right-click the “AutoScaleSettings” in the JSON Outline and delete it. We also don’t really need all those Application Insights alert rules for now, so delete those too. In fact, let’s keep it simple and also delete the “AppInsightsComponent”. We’re left with a very basic template that just creates a hosting plan and a web app. Here’s the “resources” node that has the web app and hosting plan, but not the parameters and variables.

Deployment in Visual Studio

Now let’s deploy them to Azure using Visual Studio. Simply right-click on your project and select “Deploy” and then “New…” from the drop down menu. In the form that pops up you can set your Microsoft account, your Azure subscription, the resource group you want to deploy to and the template parameters (those are stored in the *.parameters.json file). Make sure you fill out your parameter values or the deployment will fail. Now simply hit the “Deploy” button and browse to your Azure portal. If all went right you should now see a new hosting plan and a new web app in the resource group you selected.

If the resource you’re deploying already exists Azure will update that resource so it matches the ARM template. You can set literally every property you can set through the Azure portal in an ARM template as well.

A few notes considering the syntax of ARM Templates: when you need a non-static string (a parameter, variable or the result of a function) you can use the "[...]" syntax. To use a parameter you can use "[parameter('paramName')]" and to use a variable you can use "[variable('varName')]". Another very handy function is concat, which you can use like "[concat('staticstring', variable('nonstatic'))]". Other useful functions are resourceGroup() and subscription(), which can be used to get the current resource group id or location or the subscription id.

Deploy per resource

Since it’s possible to reuse hosting plans you may not want to have to deploy these together. Creating a template with only a hosting plan is easy. Simply create an empty template and add an “App Service Plan (Server Farm)” resource. The result is pretty much what we already had in the previous example, but without the web app.

So let’s create a template that only has a web app. This is a little trickier since both Azure and Visual Studio need a hosting plan with a web app. Ideally, you want to be able to set the hosting plan with your deployment since you’re probably going to have different hosting plans for different environments. Since it’s easier to start with something, simply create a template for a web app with a hosting plan.

Parameters

The first thing we’re going to do is add our own parameters. We want to be able to set the hosting plan, the resource group of the hosting plan (Azure will look in the current resource group by default), and the name of your web app because we don’t want a random name like in the previous example. I’ve given the “hostingPlanName” and “hostingPlanResourceGroup” default values because we’re most likely to use these values (for now).

Variables

The next part is where things get tricky. It’s all pretty straightforward, except for the “serverFarmId” variable, which is the hosting plan. Because the hosting plan can be in a different resource group we have to specify the complete path to the resource, including our subscription ID. While it’s possible to share resources in different subscriptions we really don’t want to go there, so I’m sticking to the current subscription.

Now try to deploy this again to a new resource group, but keep the existing hosting plan from the already existing resource group. You can now deploy this web app to any resource group using any hosting plan in any other resource group (within the same subscription).

Connecting VSTS to Azure Active Directory

Before we go any further we have to do one tiny little thing. Head over to Azure and look for the “Team Services administration”. You should be able to see your VSTS account there. Simply click your account and in the options pane that pops up click the “Connect AAD” button. This will hook up your VSTS users to Azure Active Directory (AAD). We’ll need this when we’re going to set up resource deployments to Azure in VSTS.

VSTS in Azure
VSTS in Azure

If it doesn’t work right away try going to the Azure AD first and create a directory there, then try to connect again.

Continuous Deployment using VSTS

We’re now ready to deploy our ARM templates with our web apps or just as a stand-alone release. Head over to VSTS and create a new project or use an existing one. In your project head over to the releases tab, create a new pipeline and choose for the “Empty process”.

You may want to host the examples in VSTS. I’ve used GitHub and it’s perfectly fine to connect GitHub to VSTS (although it’s not in scope of this post so you’re going to have to figure that one out yourself). It’s really easy to do though, just create a new pipeline, add an artifact and set GitHub as a source type. From there VSTS will guide you through setting up GitHub Authorization for VSTS.

Anyway, create a new release pipeline and set your artifact to the ARM templates repository. You can turn on continuous deployment (using the lightning icon in the upper right corner of your artifact). Rename your environment to “Dev” or whatever you like.

Azure Resource Group Deployment

Now it’s time to add a task. Head over to the “Tasks” page for you Dev environment and add the “Azure Resource Group Deployment” (under “Deploy”). Choose your “Azure subscription” and hit the “Authorize” button. This should work because we’ve set up VSTS in Azure AD. Leave the “Action” at “Create or update resource group”. Then pick the resource group you want to deploy to, you can either pick one from the drop down or type any name you want. Next, you have to pick the location of your resource group so either pick the location of the group you’ve picked or set the location for your new resource group.

The next thing we need to set is the “Template”. This field has a browse button for our convenience so use it and select “WebSite.json” in the “WebApp” folder so VSTS will create or update a web app. Do the same thing for “Template parameters”, but instead pick the “WebSite.parameters.json” file. In the “Override template parameters” field you can override your parameters, which you should do. The “hostingPlanName” and the “hostingPlanResourceName” should have a correct default value, but you may want to change the “webSiteName”.

VSTS release pipeline
VSTS release pipeline

Once you’ve set these you’re good to go! Give your release pipeline a name, save it, and create a new release. If all goes well your web app should be created in your Azure account.

If all goes well you can clone your Dev environment and simply overwrite your parameters (and possibly change your resource group) and create a Test environment.

Wrap up

And that’s it. It really is that easy to automatically deploy your web apps to Azure. And not only that, you can deploy VM’s, databases, queues, storage accounts, and pretty much anything you can think of, using ARM templates. If you turned on continuous deployment your deployment should start automatically when you push a commit to Git. And there’s nothing stopping you from also deploying your actual software or other Azure resources in the same deployment.

Again, don’t forget to remove your resources or you’ll be billed for them.

Happy coding!

Azure Deployment using Visual Studio Team Services (VSTS) and .NET Core

Hey all,
After three years of radio silence I decided to write a new blog post *crowd goes wild*! A lot has happened since my last post. I wrote a book on Continuous Integration, Delivery, and Deployment (with JavaScript and Jenkins, mostly); got a Microsoft Azure certificate; found a new job using those Azure skills; and I got my own company on the side. Three years ago I ended with a blog series on math, but I’m not finishing that. Apologies if you were really looking forward to #4… My new focus, for now, will mainly be Microsoft Azure, Visual Studio Team Services (VSTS), and C# .NET Core. So without further delay, let’s talk about those!

Microsoft Azure

Microsoft Azure has been around for a few years now and you’ve probably heard of it. It’s the Microsoft cloud platform that competes directly with Amazon’s AWS and Google’s Google Cloud. If you don’t really know what “the cloud” is (besides the pretty hyped buzzword) I suggest you really look into it as I believe it’s the future (heck, it’s the present)! If you do not already have an account I suggest you create one. It’s free and you only need a credit card and a Microsoft email account like hotmail or outlook.

Beware, Azure is free for now. You get free credit which is valid for one month. Some services cost a monthly fee, for others you pay per use. Just having an empty Azure account is always free. In this post we’re going to host a default web application in Azure using an App Service (which, in any case, will cost you close to nothing).

Visual Studio Team Services (VSTS)

VSTS is Microsoft’s cloud version of Team Foundation Service (TFS), which is their tool for continuous integration and deployment, source control, and agile tools like SCRUM and Kanban boards. Again, if you do not yet have a subscription I recommend you create one. Just like with Azure it’s free and you need a Microsoft email address, like hotmail or outlook. For the free version of VSTS you get quite a bit of functionality, like 240 free build minutes per month and unlimited free private Git repositories (always use Git, never Team Foundation Service Control (TFSC)). With an MSDN subscription you get some extra functionality, but the free tier is good enough for this post.

.NET Core

Do I really need to explain this? For the last few years Microsoft has focused on .NET Core, an open source multi-platform subset of the .NET Framework. While it still has some issues, even in the latest .NET Core 2.1 release, it’s also pretty cool and I’m already using it for production software. It’s a bit different than the full .NET Framework, but not a lot. You may have to download and install the latest SDK, which can all be found on the .NET Core GitHub page. Visual Studio support for .NET Core starts from Visual Studio 2017, so be sure you’re up to date, otherwise you can probably follow along with a regular .NET Framework ASP.NET Web Application.

Creating a .NET Core project

So let’s put these tools together. We start by creating a new (private) project in VSTS, which is pretty simple. In fact, you probably already created one when you created a VSTS account. I kept the default, MyFirstProject. When you browse to the Code tab of your project in VSTS you can initialize your repository (at the bottom of the page) with a README.md file and a .gitignore file (choose Visual Studio).

Initialize a repository in VSTS
Initialize a repository in VSTS.

You can copy the link or clone directly into Visual Studio on the next page in the right upper corner under the “Clone” button. I’m assuming you know how Git works, so I’m not covering that here.

Once you have the repository locally we can start by adding some code. Since we’re focusing on deploying to Azure using VSTS it doesn’t really matter what code you have, so just create an ASP.NET Core Web Application in Visual Studio 2017. You can pick a regular Web Application (which will use Razor View Pages by default) or a Model-View-Controller project, whichever you fancy. Once the default web application template loads you can commit and push it to your Git repository and we can start the fun!

Azure App Services

For the next step we’re going to mess around in Azure a bit. Log in to your Azure portal and find your App Services. Simply click the “Add” button, pick a “Web App” and hit “Create”. On the next page you can set some settings for your web app, such as the URL at which your app will be hosted (which always ends at .azurewebsites.net).

New Azure Web App
New Azure Web App

You’ll also have to create a resource group or pick an existing one. A resource group is simply a way to group certain resources for a quick overview. For example, you can have a separate resource group for dev, test, acceptance and production or for your customer portal and your internal tools, it’s all up to you.

The most important setting here is the App Service Plan/Location. This is basically a server that you pay for. A new plan is generated for you, but you can create your own as well. You have different price tiers with different hardware specifications and Azure functionality, like CPU, memory, custom domains, staging slots, and daily backups.

The location of your plan determines where the server is physically located. For me, in the Netherlands, West Europe is ideal because I know it’s in Amsterdam. Just go with the default Standard S1 plan. Even if you don’t have free credits you can delete it after you’ve read this blog post and it will only cost you like $0.03, so don’t worry about costs.

Setting up deployment from Azure

For now we’re staying in Azure. You’ve probably seen the “Build and Release” button in VSTS. That’s where you can create new build and deployment pipelines, which is great, but also a bit of work.

A build pipeline can automatically build and test your software after a push to Git. This includes restoring NuGet packages, minifying and bundling any HTML, CSS and JavaScript you might have, executing task runners and basically anything you’d ever need to make your software build and run. When your latest commit is built and tested VSTS can create an artifact for publication.

The deployment pipeline can download the artifact from a build pipeline and put it in Azure (or on your own on-premise servers). The trigger for a deployment is usually that a build has succeeded, but can also be a push to Git or a manual trigger.

When you’re building and testing your software automatically this is called Continuous Integration (or CI) and when your software is also automatically deployed this is called Continuous Deployment (or CD). There’s something in between, which holds off the deployment until a user explicitly presses a button and gives the “okay” sign, this is called Continuous Delivery. When you’re doing both CI and Continuous Delivery and/or Deployment you’re doing “CI/CD”.

Let’s set this up with pretty much a single button click. You’ve got two options in Azure. In your Web App, either configure deployment from the “Deployment options” menu item or use the newer “Deployment Center (Preview)” which, as the name suggests, is still in preview.

Deployment options in Azure
Deployment options in Azure

Deployment options

It’s really so simple I’m not even sure if I should walk you through the next steps. Click the “Deployment options” menu item and simply fill in the options you want to use.

Deployment settings in Azure
Deployment settings in Azure

And that’s that. If you now browse to the Deployment options in your Web App you’ll see that Azure now shows all your commits. If you push a new commit to your repository you can even see the build status in Azure because Azure linked it to your VSTS account.

Continuous deployment in Azure
Continuous deployment in Azure

When deployment is done, browse to your Azure website (for me that’s imunique.azurewebsites.net) and behold, your website is up and running! Try changing something on the homepage, push it to Git, wait a few minutes, and see your website get updated.

Deployment Center (Preview)

My guess is that Microsoft is planning to replace the “Deployment options” with the “Deployment Center (Preview)”. If you have already set up CI/CD with the Deployment options you’ll see your commits in here too. For the purpose of this blog we’re going to “Disconnect” (button in the top menu).

Now go to the Deployment Center (Preview) and you should see various source control options, like VSTS, Github and Bitbucket. It’s a little more visually appealing than most Azure panels. So pick VSTS and continue. Next, you have to pick a build provider. I have zero experience with Kudu, so pick “VSTS Continuous Delivery” (which somehow isn’t the default).

Deployment center in Azure
Deployment center in Azure

In the next panel you can pick your VSTS account and repository. Unfortunately, at the time of writing this blog there seems to be a bug here which prevents me from picking an account. It worked in the past and the Azure team said they have fixed the problem in an upcoming release (which is pretty often)… So try it our yourself as this may work for you!

Anyway, if you continue, Azure creates a build pipeline and a deployment pipeline in VSTS! Now that’s pretty cool. It gives you complete control over your builds and deployments as you can tweak them however you like. At the same time the Deployment Center now shows you which builds have been deployed.

Pipelines in VSTS

So browse to your VSTS and check out the build. The user interface isn’t always intuitive, but you should be able to find it. You can edit the build and click around a bit. You shouldn’t need to change it, but you can.

VSTS build
VSTS build

The deployment is a little different. There’s just one build for every piece of code, but there can be multiple deployments, for example for your entire DTAP street (dev, test, acceptance, and production). Azure created a release with a single environment, “Dev”.

VSTS release
VSTS release

You can change the deployment in the “Tasks” for that environment (either click the Dev tile or select Tasks in the upper menu). The tasks look kind of similar to the VSTS build and can be (completely) different for each environment. Mostly, your tasks will be more or less the same and you’ll only have some values that differ per environment. You can use “Variables” for these different values, but that’s out of the scope of this post.

It’s also pretty easy to clone an entire environment (including it’s tasks and variables). Just hover over the tile of the environment you’d like to clone and a “Clone” button will show up. Click it and you get a new environment that’s exactly like the one you cloned. Further setup of builds and releases is out of scope of this post, but you can click around if you like.

When you’re done playing around simply delete the release and the build pipelines (in that order). Also make sure you disconnect the deployment in the Azure Deployment Center.

Deployment from Visual Studio 2017

Finally, let’s switch back to Visual Studio. You can do the exact same thing in Visual Studio. Right click on the project you want to release (which is your web project) and choose “Overview” in the drop down menu. In the page that opens go to “Publish” (in the menu on the left).

Here you can publish or set up your CI/CD. Click the “Start” button under “Continuous Delivery” and you can once again choose a VSTS subscription and an App Service (it creates a new App Service by default, but you can also select an existing one). Simply clicking the “Ok” button will have the same outcome as what we did in the Deployment Center. It will create a new build and deployment pipeline that is also visible in your Deployment Center. Pretty cool!

Going back to the Publish page in Visual Studio, you can also just publish directly. This is pretty useful if you want to release something to Azure right now. Simply click the “Start” button under “Publish”. You can now either create a new Azure App Service or pick an existing one. Just follow the wizard and your application will be published to Azure directly, without a build or deployment pipeline. Be careful though, your local build will be deployed, which will often be different than a build from the build server.

Wrap up

So there you have it! Deployment was never this easy! In just a couple of minutes we’ve created an application, put it in source control, and deployed it to the cloud, ready for the world to see (you can restrict access, but that’s out of scope). In the past it could take you hours to set up a new server (or days and even weeks if you still had to order one), but this alternative is a lot faster and cheaper.

The next step is using ARM (Azure Resource Management) templates to automatically create App Services and other Azure resources. Check out my post on ARM templates to get started.

Don’t forget to delete your App Service and your App Service plan or you’ll be charged for them!

Good luck and happy coding!

Maths in IT #3: Algebra of sets

Hey everyone and welcome back! I’ve already announced this posts subject in my previous post, algebra of sets. So algebra is difficult and scary. At least that’s what people say (hmm hmm), so let’s shake that off. I’m sorry if you got the reference 🙂

If you haven’t read my previous posts I really suggest you do so as this post will continue where the previous one left off.

  1. Maths in IT #1: Basic set theory
  2. Maths in IT #2: Venn diagrams
  3. Maths in IT #3: Algebra of sets
  4. Coming soon…

Here’s a quick cheat sheet of symbols introduced in the previous post:
Intersection: A \cap B = \{x | x \in A \land x \in B\}
Union: A \cup B = \{x | x \in A \lor x \in B\}
Universe U: The collection of all elements we consider for our scenario.
Absolute complement (or simply complement): A^c = \{x | x \in U \land x \notin A\}
Relative complement of A in B: B \cap A^c = A \setminus B

Algebra

What exactly is algebra? It’s about the study of symbols, or operations, and laws for manipulating these operations. Elementary algebra, for example, is a set of laws for operations (+, -, /, *) on numbers. Just like that \cap, \cup and A^c are operations on sets and the laws used to manipulating them is called the algebra of sets.

These laws can help us to simplify formulas. So let’s get started.

Commutative property

Consider the following addition: 2 + 3. I’m pretty sure we all know the answer is 5 and the answer doesn’t change if we switch the numbers to 3 + 2. More abstract we can say that for every two numbers x and y goes that x + y = y + x.
When the order of operands (that is the objects on which an operation is performed) in an operation doesn’t matter for the outcome of that operation we call that operation commutative.

Multiplication (on real numbers) is commutative too, because 2 \cdot 3 = 3 \cdot 2 (you may be used to seeing x or * as multiplication symbol, but you should remember \cdot from high school). Minus isn’t commutative, because 3 - 2 \neq 2 - 3 and neither is division, 2/3 \neq 3/2.

When working with sets we can say that A \cap B = B \cap A and A \cup B = B \cup A. That means both \cap and \cup are commutative.

Associative property

When the order in which the same operation is performed in a single formula is unimportant for the outcome of that formula we say that the operation is associative. For example (1 + 2) + 3 = 1 + (2 + 3). It doesn’t matter if we first evaluate 1 + 2 and then add 3 or first evaluate 2 + 3 and add it to 1, the outcome is always 6.
The same holds true for multiplication, because (2 \cdot 3) \cdot 4 = 2 \cdot (3 \cdot 4) but not for subtraction, (10 - 8) - 2 \neq 10 - (8 - 2) and division, (16/4)/2 \neq 16/(4/2).

When looking at sets we find that both \cap and \cup are associative, because (A \cap B) \cap C = A \cap (B \cap C) and (A \cup B) \cup C = A \cup (B \cup C).

That means parenthesis can be omitted, removing any confusion on operator precedence.

And with the commutative and associative properties of both intersection and union we can now simplify
C \cap (D \cap A) \cap B
to A \cap B \cap C \cap D
or C \cup (D \cup A) \cup B
to A \cup B \cup C \cup D.
And when we combine intersection and union we can simplify as well. D \cap (C \cup (B \cup A)) can be simplified to (A \cup B \cup C) \cap D. Notice that we still need parenthesis to define the order of the \cap and \cup operations.

We can now also take the intersection or union of any number of sets because we don’t have to worry about the order of operands or operators. A_1 \cap A_2 \cap A_3... gets tiresome really quick though. So suppose we have A_1 to A_{100}. The intersection or union of all those sets can now be expressed as follows:
\displaystyle\bigcup_{i=1}^{100} A_i     or     \displaystyle\bigcap_{i=1}^{100} A_i
And inline this look like \cup_{i=1}^{100} A_i and \cap_{i=1}^{100} A_i.

Sometimes we don’t know if a set has 100 sets. Especially when a set is an infinite set of sets the above notation is not feasible. We can now use index notation to intersect or union all sets within the set.
\displaystyle\bigcap_{i \in I} A_i     or     \displaystyle\bigcup_{i \in I} A_i
\cap_{i \in I} A_i and \cup_{i \in I} A_i basically means “intersect/union for every set i in set I” (think of a foreach loop on a List of Lists in C#).

Distributive property

The distributive property is difficult to describe in words. Just remember that multiplication is distributive over addition, meaning that 3 \cdot (4 + 5) = (3 \cdot 4) + (3 \cdot 5). More generally x(y + z) = xy + xz (the multiplication symbol is often omitted between letters).

\cap is distributive over \cup, meaning that
A \cap (B \cup C) = (A \cap B) \cup (A \cap C)

Likewise \cup is distributive over \cap, so
A \cup (B \cap C) = (A \cup B) \cap (A \cup C)

We can now show that (A \cap B) \cap (A \cup B) = A \cap B. When we apply the distributive property on (A \cap B) \cap (A \cup B) we get ((A \cap B) \cap A) \cup ((A \cap B) \cap B).
Here it is again, but with colors (and non-mathematical notation) so you can follow where everything went:
(A n B) n (A u B) = ((A n B) n A) u ((A n B) n B).
Applying commutativity and associativity we can now simplify further:
((A \cap B) \cap A) \cup ((A \cap B) \cap B) = (A \cap A \cap B) \cup (A \cap B \cap B).
In the previous article, Maths in IT #2: Venn diagrams, we have seen that A \cap A = A and A \cup A = A. This is called idempotence and can be applied here (twice) to simplify further:
(A \cap A \cap B) \cup (A \cap B \cap B) = (A \cap B) \cup (A \cap B) = A \cap B

Let’s repeat that:

(A \cap B) \cap (A \cup B)

Distributivity

((A \cap B) \cap A) \cup ((A \cap B) \cap B)

Associativity

(A \cap B \cap A) \cup (A \cap B \cap B)

Commutativity

(A \cap A \cap B) \cup (A \cap B \cap B)

Idempotence

(A \cap B) \cup (A \cap B)

Idempotence

A \cap B

So we could make use of distributivity, associativity, commutativity, and idempotence to make a rather difficult formula pretty easy! Unfortunately it was a lot less easy to do…

Other properties

Unfortunately we’re not done yet. There are a few more properties, some of which we’ve already encountered, that I’d like to go over.

First we have the absorption law, which states that A \cap (A \cup B) = A and A \cup (A \cap B) = A.

We’ve seen the so called null element, or empty set, \emptyset. Just like with numbers (x + 0 = x, x - 0 = x and x \cdot 0 = 0) there are some rules for working with \emptyset. Luckily they’re not that hard.
A \cap \emptyset = \emptyset and A \cup \emptyset = A.

When a universe U is defined we can say the following:
A \cap U = A and A \cup U = U.

I’ve shown you that a double complement returns the original: (A^c)^c = A.
Also, A \cap A^c = \emptyset and A \cup A^c = U.

De Morgan’s Law

Finally we’ll look at the law of De Morgan, named after its inventor, the British mathematician Augustus De Morgan.
Suppose my blog can only be read by people who have not studied maths or IT. Let A = \{x | \text{x has studied maths}\} and let B = \{x | \text{x has studied IT}\}. So now the people who may read my blog are people who haven’t studied maths or IT: (A \cup B)^c. I could also say people who haven’t studied maths and people who haven’t studied IT may read my blog: A^c \cap B^c. But that’s the same set of people!
In the following Venn diagram this is represented by the white part.

A Venn diagram of De Morgan's Law
A Venn diagram of De Morgan’s Law

So in short (A \cup B)^c = A^c \cap B^c.

Now suppose my blog may be read only by people who haven’t studied both maths and IT. So that’s (A \cap B)^c. Or we could say people who haven’t studied maths and who haven’t studied IT, A^c \cup B^c. And again this is the same set of people. In the Venn diagram it’s the part where A and B don’t overlap (so almost everything).

A Venn diagram of De Morgan's Law
A Venn diagram of De Morgan’s Law

And so that means (A \cap B)^c = A^c \cup B^c.

Now let’s take another example of how to simplify a formula. Suppose we have B \cap (A \cap B)^c. So let’s first apply De Morgan’s law: B \cap (A \cap B)^c = B \cap (A^c \cup B^c). And now you may recognize a nice pattern for distributivity! So let’s apply distributivity, B \cap (A^c \cup B^c) = (B \cap A^c) \cup (B \cap B^c). That’s nice, because we know that B \cap B^c = \emptyset. And now we know that (B \cap A^c) \cup \emptyset = B \cap A^c. And maybe you’ll remember this from my previous post, this is B \setminus A. And there you have it!

Again, let’s repeat:

B \cap (A \cap B)^c

De Morgan’s law

B \cap (A^c \cup B^c)

Distributivity

(B \cap A^c) \cup (B \cap B^c)

Complement rule

(B \cap A^c) \cup \emptyset

Null element

B \cap A^c

Relative complement

B \setminus A

And there you go! You algebra master, you!

So how will this help you in your day to day programming tasks? For most of us very little I’m afraid, except that you can work out some difficult set operations on paper and possibly simplify requirements. I’m no database developer, but I can imagine you’d need this when developing databases (and to a lesser extent when working with databases).

We did venture into the world of algebra though, and if you’re still with me I say congratulations! Algebra is a bit too abstract and difficult for many people.

I’m leaving you with a little cheat sheet of today’s lessons and I hope to see you again next time (no algebra, I promise)!

Commutativity:
A \cap B = B \cap A
A \cup B = B \cup A

Associativity:
(A \cap B) \cap C = A \cap (B \cap C)
(A \cup B) \cup C = A \cup (B \cup C)

Distributivity:
A \cap (B \cup C) = (A \cap B) \cup (A \cap C)
A \cup (B \cap C) = (A \cup B) \cap (A \cup C)

Idempotence:
A \cup A = A
A \cap A = A

Absorption:
A \cup (A \cap B) = A
A \cap (A \cup B) = A

Null element:
A \cap \emptyset = \emptyset
A \cup \emptyset = A

Universe:
A \cap U = A
A \cup U = U

Complement:
A \cap A^c = \emptyset
A \cup A^c = U

Double complement:
(A^c)^c = A

Relative complement:
A \cap B^c = A \setminus B

De Morgan’s Law:
(A \cap B)^c = A^c \cup B^c
(A \cup B)^c = A^c \cap B^c

Maths in IT #2: Venn diagrams

Hey everyone, welcome back to part two of the Maths in IT series. I got a lot of positive response, so I guess I should just keep doing what I was already doing. This post will continue where part one left off, so if you haven’t read it I suggest you do so now before continuing.

  1. Maths in IT #1: Basic set theory
  2. Maths in IT #2: Venn diagrams
  3. Maths in IT #3: Algebra of sets
  4. Coming soon…

Here’s a quick cheat sheet with symbols I’ll use in this article:
Explicit definition: A = \{a, b, c\}
Implicit definition: A = \{x | \text{ x is a letter in the alphabet}\}
a is an element of A: a \in A
a is not an element of A: a \notin A
A is a subset of B: A \subset B
Empty set: \emptyset

Venn diagrams

As promised we’ll use this post to combine sets. Before we do that let’s take a look at how to visually represent collection. We can do this using a Venn diagram. A Venn diagram is about as simple as it gets (although they can be pretty complex too). Each set is represented by a circle. The diagram can illustrate relationships between the represented sets.

Let’s look at an example. We have two collections, A = \{a, b, c\} and B = \{d, e, f\}. Let’s show them in a Venn diagram.

A Venn diagram
A Venn diagram

When two sets have no shared elements, or for every x \in A goes that x \notin B, we say the sets are disjoint.

Now suppose A \subset B (A is a subset of B). We can show this in a Venn diagram and you’ll recognize it immediately.

A Venn diagram of a subset
A Venn diagram of a subset

When A is a subset of B then B overlaps A completely.

In the next sections we’ll combine sets and use Venn diagrams to visualize what elements we’re interested in.

Intersection

So suppose we have two collections, A = \{a, b, c, d\} and B = \{c, d, e, f\}. If I asked you which elements are in both A and B you’d answer c and d.

This is called the intersection of A and B.
We can write this as A \cap B.
In this example we can say A \cap B = \{c, d\}
More formally we say that A \cap B = \{x | x \in A \text{ and } x \in B\}.
And the symbol for “and” is actually \land, so to formalize it completely:

A \cap B = \{x | x \in A \land x \in B\}

Phew, that looks a whole lot like maths! You should read that as “the intersection of A and B is the collection of every x where x is an element of A and x is an element of B.”
And here is the Venn diagram, which visualizes this nicely. The intersection is the part in the thick black line where A and B overlap.

A Venn diagram of an intersection
A Venn diagram of an intersection

With an intersection we can give a formal definition of disjoint sets. When A \cap B = \emptyset then A and B are disjoint.

Furthermore we can say that for any collection A goes that A \cap \emptyset = \emptyset.
Also A \cap A = A.
And when A \subset B then A \cap B = A (check the subset Venn diagram).

Union

The intersection of two sets is the set of elements x where x is in A and B. Likewise, the union of two sets is the set of elements that are in set A or B. So when we have A = \{a, b, c, d\} and B = \{c, d, e, f\} then the union of A and B is \{a, b, c, d, e, f\} (no doubles).

We can write a union of A and B as A \cup B.
Like \land is the symbol for “and” \lor is the symbol for “or”. So the formal definition of union is as follows:

A \cup B = \{x | x \in A \lor x \in B\}

Read that as “the union of A and B is the collection of every x where x is an element of A or x is an element of B.”
In a Venn diagram the union is basically just both sets (the part in the thick black line).

A Venn diagram of an overlapping union
A Venn diagram of an overlapping union

Unlike an intersection, a union of disjoint sets is not an empty set (notice that both sets have a thick black line).

A Venn diagram of a union
A Venn diagram of a union

Now for any collection A goes that A \cup \emptyset = A.
And, again, A \cup A = A.
Also when A \subset B then A \cup B = B (check the subset Venn diagram).

Universe and Complement

When we’re talking about sets we’re usually not talking about that set in isolation. When I tell you that non-smokers live longer you understand that they live longer compared to people that do smoke. And you also understand that non-smokers and smokers combined make up for the worlds population. In this case we’re saying that we have a set of non-smokers in a universe of all people. We denote a universe with the capital letter U.

For every set A_1, A_2, A_3... A_n goes that A_n \subset U.
That makes sense as U represents all elements we wish to consider for our case. No collection can ever contain an element that is not a part of all elements.

In a Venn diagram we draw a universe as a rectangle in which all our sets are drawn. In the following example N is the set of non-smokers and U is the universe containing all people.

A Venn diagram of a universe
A Venn diagram of a universe

Now with the notion of a universe we can say we want all elements that are not in any collection A. We call this the complement of A and we write it as A^c. In the following Venn diagram the white part represents A^c.

A Venn diagram of a complement
A Venn diagram of a complement

We can now formally define the complement:

A^c = \{ x | x \in U \land x \notin A \}

It goes without saying, but \emptyset^c = U and U^c = \emptyset.
A little less obvious is that (A^c)^c = A. It makes sense though, as we first take everything that isn’t A (the complement of A) and then we take everything that isn’t in the resulting set, but that is A. Try drawing it in a Venn diagram and you’ll see what I mean.

A complement relative to a universe is called an absolute complement. A complement can also be relative to other sets. For example, when no universe is defined and we have sets A and B then the relative complement of A in B is the set of elements of B that are not in A. This takes the form of B \cap A^c or A \setminus B.

A \setminus B = \{ x | x \in B \land x \notin A \}

In a Venn diagram A \setminus B look as follows:

Venn10

Combining sets

We can now combine sets using intersection, union and complements. For example, let’s say our universe U is all living creatures on earth. Within U we have sets M, containing all mammals, B, containing all birds, and E, containing all animals that lay eggs. Formally:
U = \{ x | \text{x is an animal} \}
M = \{ x | x \in U \land \text{x is a mammal} \}
B = \{ x | x \in U \land \text{x is a bird} \}
E = \{ x | x \in U \land \text{x lays eggs} \}
Giving the following Venn diagram we can already draw some conclusions.

A Venn diagram with multiple sets
A Venn diagram with multiple sets

We can see that all birds lay eggs. Some mammals lay eggs too. No animal is both a bird and a mammal. As you see Venn diagrams can be really useful.
Now suppose we want the set of all mammals that lay eggs and also all birds. This is the collection (M \cap E) \cup B. That is, we take the intersection of M and E and union the result with B. In a Venn diagram we can see this collection (the red colored parts).

A Venn diagram with multiple sets
A Venn diagram with multiple sets

Now we can get every combination of sets using intersection, union and complement. It’s not always easy, but it’s possible.

Some code

As promised I’m going to keep things practical. So what’s the practical use of all this? Well, most languages let you work with exactly these functions!

For example, take a look at this SQL expression.

(SELECT 1
UNION SELECT 2
UNION SELECT 3)
INTERSECT
(SELECT 3
UNION SELECT 4)

What will that return? It returns only 3 because 3 is an element of both collections. This example also shows the UNION operator. Basically this is the set (\{1\} \cup \{2\} \cup \{3\}) \cap (\{3\} \cup \{4\}).

And SQL knows complement too.

(SELECT 1
UNION SELECT 2
UNION SELECT 3)
EXCEPT
(SELECT 3
UNION SELECT 4)

What happens here is that we have (an implied) universe U = \{1, 2, 3, 4\} and EXCEPT is the complement. So this query is basically the formula \{3, 4\}^c.
We could also say that U is not defined and this is the relative complement of \{3, 4\} in \{1, 2, 3\}: \{3, 4\} \setminus \{1, 2, 3\} = \{1, 2\}.

But what about C#? In my previous post we’ve seen HashSet<T>, but I’m not going to use that now. Instead I’m just going with a List<T> as LINQ provides various extension methods for working with sets. Notice that HashSet<T> has its own methods in addition to the LINQ methods.

List<int> a = new List<int>();
a.Add(1);
a.Add(2);
a.Add(3);

List<int> b = new List<int>();
b.Add(3);
b.Add(4);

List<int> intersect = a.Intersect(b).ToList();
List<int> union = a.Union(b).ToList();
List<int> except = a.Except(b).ToList();

You may wonder what that looks like in Haskell (since I’ve shown you Haskell the last time too), but it’s not really that different.

Prelude> import Data.List
Prelude Data.List> [1, 2, 3] `intersect` [3, 4]
[3]
Prelude Data.List> [1, 2, 3] `union` [3, 4]
[1,2,3,4]
Prelude Data.List> [1, 2, 3] \\ [3, 4]
[1,2]

I’ve actually needed this stuff in my day to day work. It’s not that hard and sometimes it’s an explicit business requirement. For example I needed all sales orders from The Netherlands, Belgium and Luxembourg (the BeNeLux). That’s really just a union! And I’ve needed intersect too, give me all Dutch customers that are not in some list of customers. And how about all non-Dutch customers? That’s just the complement!

Next time I’d like to continue with algebra and algebra of sets in specific. Does that sound like it will give you nightmares? Don’t worry, I’ll be gentle!

See you next time!

Maths in IT #1: Basic set theory

Welcome back everyone. Today I have a little something different for you. No new language or framework, but something that’s been around for millennia: maths.
When asking programmers about maths you’ll find two kinds of people, those who say you don’t need maths to be a good programmer and those who say maths is essential. Personally I think both are true. For some applications and industries you really don’t need advanced maths, but go into robotics, machine learning, statistics, or that kind of thing and you’re going to need maths, lots of it. And whether you need it or not, computers, programming languages and databases all wouldn’t exist without maths.
For now, let’s put it this way: knowing a thing or two about maths gives you an edge as a programmer!

Maths is everywhere: physics, chemistry, biology, economy and, yes, even in the arts! For this series I’ll focus on the maths we need in IT. I don’t know how much entries it’s going to have or what I’ll be discussing (and what I won’t be discussing), but I’ll be sure to keep it somewhat practical. No prior maths knowledge is assumed. Don’t worry, I’ll still post about code once in a while too!

  1. Maths in IT #1: Basic set theory
  2. Maths in IT #2: Venn diagrams
  3. Maths in IT #3: Algebra of sets
  4. Coming soon…

Collections

Do I need to tell you why we should study collections? You probably use them in your code every day in the form of arrays, database tables, lists or hash tables. What you probably didn’t know is that collections have a lot of mathematical theory! Since collections are very important in both maths as in programming I’m going to start this series here. More specifically we’re going to talk about sets.

A set is an unordered collection containing only distinct values.

Let’s look at an example of a set in real life. Do you collect anything? Maybe you collect old records, each record in your collection can be uniquely identified and doubles are for trading or selling. Also, no matter in which order you put the records on the shelf, it’s still the same collection of records. So a record collection is really a set.

Likewise we can have a collection of paintings or stamps. Another collection is an alphabet, for example the English, Russian or Greek alphabet. Using these alphabets we can construct a language. We have natural languages (like the ones I just mentioned) and formal languages. Programming languages like C#, Java, C or Haskell are examples of formal languages.

A collection in maths is usually indicated by a single suggestive capital letter, such as R for records, S for stamps or A for alphabet. If we have more than one collection we can use index notations to uniquely identify them: A_{1}, A_{2}, A_{3}

The notation for a single collection containing elements a, b and c is \{a, b, c\}. This is called the explicit definition. We can now declare a collection A as A = \{a, b, c\}.
And of course we can have a collection of collections: C = \{\{a, b\}, \{c, d\}, \{a, c\}\} . Notice that collection \{a, c\} is unique even though a and c are already elements in other collections.
The following set of sets isn’t valid: \{\{a, b\}, \{b, a\}\}. Because the order of elements in sets is ignored this set contains the same set twice, but sets also must have distinct values.
Now let’s say we have English alphabet E, Russian alphabet R and Greek alphabet G. The collection of alphabets is written as \{E, R, G\}.

Now we want to indicate that a is an elements of E. We do this using the following syntax: a \in E.
To indicate that \lambda (the Greek letter lambda) is not an element of E we use notation: \lambda \notin E.

We can compare collections. Collections are considered to be equal when they contain exactly the same elements, no more and no less. \{a, b, c\} = \{a, b, c\}, \{a, b, c\} = \{c, b, a\} (remember, order is ignored), \{a, b, c\} \neq \{d, e, f\} and \{a, b, c\} \neq \{a, \{b\}, c\} (the collection \{b\} does not equal b).

So far we’ve seen only explicitly defined collections. We can also implicitly define collections. This is especially useful when a collection contains too many elements to write down. For example a collection containing all countries on Earth. The notation for such a collection is as follows: \{x | \text{x is a country on Earth}\}. You should read that as “the collection consisting of all (objects) x for which x is a country on Earth”.
Generally we can say that an implicitly defined collection is in the form of \{x | P(x)\} where, in this example, P(x) is the statement that x is a country on Earth. Actually P(x) is a function taking parameter x and returning whether x is or isn’t a part of the set. We’ll look at functions in a later blog post.

To indicate how many elements are in any given (finite) collection we can use notation |A|. So |\{a, b, c\}| = 3 and |E| = 26 (where E is the English alphabet). Of course this is only possible when our collection is finite (we can count the elements).

If a collection is empty (it contains no elements) or some collection A = \{\} we can use a special symbol \emptyset. And of course |\emptyset| = 0 (an empty set has 0 elements).

When we allow the same element to appear in a collection more than once and we start taking the order of elements into consideration we’re speaking of a row. The syntax for a row is simply putting the elements next to each other. For example, using the set \{a, b, c\} we can make the rows a, ababc, baac, caab, but not baad (because d is not in the set).

Infinite collections

So far we’ve looked at finite collections.  Let’s look at some infinite collections now. Consider the collection of all numbers: 1, 2, 3, 4… In theory we can always add 1 to any number, so we can never stop counting. A few of these collections are so important that they got their own symbol.
First we have the natural numbers: \mathbb{N} = \{0, 1, 2, 3, ...\}.
If we add negatives to the collection we get all whole numbers, or integers: \mathbb{Z} = \{..., -3, -2, -1, 0, 1, 2, 3, ...\}.
If we also allow fractions, like \frac{1}{2} (a half) or \frac{1}{4} (a quarter) then we have the collection of rational numbers \mathbb{Q}.
Not all numbers can be expressed in fractions, for example pi (\pi, the surface of a circle with radius 1) or \sqrt{2}. When we want to include those numbers we get the collection of real numbers \mathbb{R}.

Now suppose we want all positive natural numbers, so 1, 2, 3… (excluding 0). We can indicate this with \mathbb{N}^+. Likewise, all negative numbers -1, -2, -3… can be indicated using \mathbb{Z}^-. And of course we can use \mathbb{Q}^+, \mathbb{Q}^-, \mathbb{R}^+ and \mathbb{R}^- to indicate positive or negative fractions and real numbers too.

And we can now define new infinite collections using implicit definitions. The collection of all even numbers, for example, can be defined as follows: E = \{ x | x \in \mathbb{Z} \text{ and x is even}\}.
So here E is a collection consisting of all x for which x is an element in \mathbb{Z} (all integers) and x is even.

Subsets

If a collection A contains elements that are all elements of another collection B we say that A is a subset of B. For example A = \{a, b, c\} is a subset of B = \{x | \text{x is a letter in the English alphabet}\} because a, b and c are all letters in the English alphabet.

More formally we can say that if A and B are both collections and for every x \in A applies x \in B then A is a subset of B.

We can write this as A \subset B. Since, in the previous example, A does not contain all letters in the English alphabet B we can also say that B is not a subset of A, this is written as B \not\subset A.

Given the above definition we can also say that A \subset A or A is a subset of itself. The empty collection \emptyset is a subset of all collections (including itself).

When a collection A \subset B, but A \neq B then we say that A is a proper subset of B. We may write this as A \nsubseteq B.
Sometimes you may see the notation A \subseteq B to indicate that A is a subset of B that may or may not be equal to B.
I’ll simply use A \subset B throughout the series).

When A \subset B and B \subset A then A = B.

In a similar manner we can say that B is a superset of A if A is a subset of B. This is notated as B \supset A (it’s the subset-symbol reversed). And of course we can say B \supseteq A to indicate that B is a superset of A that may or may not be equal to A. For a proper superset we may use A \nsupseteq B notation.

And when A \supset B and B \supset A then A = B.

Some code

I promised I’d keep this series somewhat practical. So let’s look at some code. Consider the following C# sample using your favorite .NET collection class.

List<int> list = new List<int>();
list.Add(1);
list.Add(2);
list.Add(3);
list.Add(1);
int thirdItem = list[2]; // 0-based index.

The list now contains the items 1, 2, 3 and 1 again.  We can also get the item at the nth position, which is only useful when we know the order of the elements within the list. Based on that we can conclude that the List class in .NET is not a set. If we want a set in .NET we can use the HashSet class instead.

HashSet<int> set = new HashSet<int>();
set.Add(1);
set.Add(2);
set.Add(3);
if (!set.Add(1))
{
    Console.WriteLine("Item 1 couldn't be added.");
}
//int secondItem = set[1]; // Doesn't compile.

Because a set only contains unique items a hash of each item can be generated which means that lookup time for sets is much faster than that of lists (especially when the size of the list grows). The HashSet<T> can be compared to the Dictionary<TKey, TValue> class, but without values.

Unfortunately C# doesn’t know list generators. Working with infinite collections isn’t very do-able in C# either. Let’s say you’d like to get all positive even numbers, E^+ = \{ x | x \in \mathbb{N}^+ \text{ and x is even}\}. We have a few problems. First \mathbb{N} isn’t available in C#. We could take Int32.MaxValue (but that threw an OutOfMemoryException). So let’s just take all positive evens smaller than or equal to a million.

IEnumerable evens = from x in Enumerable.Range(1, 1000000)
                         where x % 2 == 0
                         select x;
List evaluatedEvens = evens.ToList();

Something like that is really the closest we can get in C# without going through a lot of trouble.

And at this point I stand corrected. Paulo Zemek pointed out that it’s actually pretty easy to work with infinite collections by utilizing the yield keyword. The next (Console Application) example illustrates this (although int will eventually overflow…).

static void Main(string[] args)
{
    foreach (int i in Evens().Take(10))
    {
        Console.WriteLine(i);
    }
    Console.ReadKey();
}

static IEnumerable Evens()
{
    return XToInfinity(1).Where(i => i % 2 == 0);
}

static IEnumerable XToInfinity(int start)
{
    int current = start;
    while (true)
    {
        yield return current;
        current++;
    }
}

That’s still a lot of typing though…

Let’s take another language that solves these kinds of problems a little better, a language that is a little closer to actual maths, Haskell.

evens = [ x | x <- [1..], x `mod` 2 == 0]

And there you have it. Notice that the code is actually pretty close to the mathematical notation. Take each x where x is an element of [1..] (one to infinity) and where x is even.
We can easily take the first 10 items of this infinite collection without crashing the program or encountering out of memory exceptions.

*Main> take 10 evens
[2,4,6,8,10,12,14,16,18,20]

This isn’t a blog on Haskell, so I can’t really expand on what it does, but I can say that Haskell is a “lazy” language, which means it doesn’t evaluate the items in the list until you actually need them. In this case it will only evaluate the first ten items of evens, keeping it from hogging our memory and CPU.

That’s it for now. We’ve had a very basic introduction to sets, but if you’re not familiar with this stuff it gets hard pretty quickly. So let’s take it nice and easy. Next time we’re going to combine collections and take a look at the Venn diagram.

Hope to see you next time!

Apply, Call and Bind on JavaScript functions

I’m a bit short on time this week so I’ll keep it short. Last week I talked about Prototype in JavaScript, a fundamental JavaScript feature that too few programmers know about. Today I’d like to talk about a few more features that people either don’t know about or are confused about, apply(), call() and bind().

As usual I have the examples on GitHub in the apply-call-bind repository.

Also, in case you missed it, I have a free Udemy course on ReactJS and Flux for you. Only 50, so you best be quick to get it!

Function invocation

So let’s talk a bit about functions and how they can be invoked. Let’s look at a simple function with a simple invocation.

var sum = function (a, b) {
    return a + b;
};
var x = sum(1, 2);
console.log(x);

It doesn’t get simpler than this and x will have the expected value of 3. Let’s make that slightly less simple. We want to be able to add an undefined number of numbers! So how would we handle that? How about this?

var sum = function (numbers) {
    var result = 0;
    numbers.forEach(function(n) {
        result += n;
    });
    return result;
};
var x = sum([1, 2, 3, 4]);
console.log(x);

That looks good, right? Unfortunately, invoking the function is now more difficult as we always need an array. What if I told you there is another way?

Remember that this is JavaScript and JavaScript is, well, weird. We can call a function with any number of input parameters we want. Too few and our function might fail, too much and the extra parameters will be ignored.
There is sense in all this though. Within each function you can inspect the input parameters through an array-like object, arguments.

Let’s rewrite the first version of sum, but using the arguments list this time.

var sum = function () {
    return arguments[0] + arguments[1];
};
var x = sum(1, 2);
console.log(x);

But if we can do that we can use ANY number of parameters! Arguments is array-like, which mean we can loop through the values (it doesn’t have a forEach method though).

var sum = function () {
    var result = 0;
    var i = 0;
    for (i; i < arguments.length; i++) {
        result += arguments[i];
    }
    return result;
};
var x = sum(1, 2, 3, 4);
console.log(x);

That’s pretty nice! We can now use the same simple syntax for any number, or a variable number, of input parameters. So now I’m going to be a real pain in the ass… We got this nice function that takes any number of parameters in the plain and simple syntax we’re used to, but… I want to pass in an array!

Apply

If I were your manager you’d be annoyed at best. I asked for plain and simple syntax, you deliver, and now I want an array after all!? Worry not! Your function is awesome and it can stay. What we’re going to use is a function on the function prototype (meaning all function objects have this function defined), apply.

So that sounded weird… Functions have functions? Yes they do! Remember that a function in JavaScript is just an object. To give you an idea, this is how it looks.

myFunc(); // Invocation of a function.
myFunc.someFunc(); // Invocation of a function on a function.

So let’s look at this function, apply, which allows you to pass input parameters to any function as an array. Apply has two input parameters, the object on which you want to invoke the function (or the object this points to in the function) and the array with input parameters to the function.

So let’s invoke our function, sum, using apply. Notice that the first parameter can be null as we’re not invoking sum on an object.

var x = sum.apply(null, [1, 2, 3, 4]);
console.log(x);

So that’s awesome, right? Let’s look at an example that doesn’t use the arguments variable.

var steve = {
    firstName: 'Steve',
    lastName: 'Ballmer',
    doThatThing: function (what) {
        console.log(this.firstName + ' ' + this.lastName
        + ': ' + what + '! ' + what + '! ' + what + '!');
    }
};
steve.doThatThing.apply(null, ['Developers']);
steve.doThatThing.apply(steve, ['Developers']);

So in this example the function doThatThing is just a regular function with a regular named input parameter and we can still invoke it using apply. In this case we also need to pass in the first parameter to set this. If we don’t specify the first parameter firstName and lastName in the function will be undefined.

We can also put in another variable as first input parameter.

var sander = {
    firstName: 'Sander',
    lastName: 'Rossel'
};
steve.doThatThing.apply(sander, ['Blogs']);

That’s a bit weird, isn’t it? Even though Sander does not have the doThatThing function we can invoke it as if Sander did. This can be very useful behavior as we’ll see in the next section!

Call

Another function, which looks like apply, is call. Call allows you to simply invoke a function, but makes you specify the object which serves as this within the function, much like apply. The only difference between apply and call is the manner in which input parameters are supplied. Apply needs an array with values, call needs the values separated by a comma (like regular function invocations).

steve.doThatThing.call(null, 'Developers');
steve.doThatThing.call(steve, 'Developers');
steve.doThatThing.call(sander, 'Blogs');

So why would you want to do this? It seems it only obfuscates the code… Consider the following scenario, you have some function that takes a callback as input and invokes it.

var someFunc = function (callback) {
    console.log('this is ' + this);
    callback('This');
};
someFunc(steve.doThatThing);

What will it print? Undefined undefined… Sure, when doThatThing is invoked as callback() this will be the global Window object. We have three options now… I’ll discuss two now and I’ll give the third in a minute.
So first we could make it so that this is not used in the callback. For this we’d need to create a closure, or a new function that invokes doThatThing on steve as if it were a normal function.

someFunc(function (what) {
    steve.doThatThing(what);
});

It works, but its a lot of bloated code, so we don’t really want that. So there’s a second option, we allow the this object to be passed to the someFunc as an input parameter. And then we can invoke the function on that object using call (or apply)!

var someFuncThis = function (callback, thisArg) {
    callback.call(thisArg, 'This');
};
someFuncThis(steve.doThatThing, steve);

A lot of JavaScript libraries and frameworks, like jQuery and Knockout.js, use this pattern, passing in this as an optional input parameter.

Bind

If you’ve been paying attention you’ll remember I just said there was a third method to solve our little problem with this in the callback function. Next to apply and call functions also have a bind function. Bind is a function that takes an object as input parameter and returns another function. The function that is returned invokes the original function with the input object as this context. Let’s just look at an example.

someFunc(steve.doThatThing.bind(steve));
someFunc(steve.doThatThing.bind(sander));

And it even works if your original this isn’t the function itself.

var f = steve.doThatThing;
f = f.bind(steve);
someFunc(f);

You’ll need this when passing JavaScript native functions as callbacks to third party libraries.

someFunc(console.log);
someFunc(console.log.bind(console));

The first call fails with an “Illegal invocation” because this inside the log function has to be console, but it isn’t if it’s invoked as callback. The second line works as we’ve bound this to console.

So here are three functions that all invoke a function slightly differently than what you’re used to. All three are indispensable when you’re doing serious JavaScript development though.

If you’d like to know more about functions and how they work in JavaScript I can recommend JavaScript Succinctly.
We’ve also seen a lot of trouble with the this keyword. If you’re wondering why, you should check out this six day course by Derick Bailey: The Rules For Mastering JavaScript’s “this”. I can highly recommend it!

This blog is a bit shorter than usual, but, as they say, it’s quality over quantity. Hope to see you back again next week!

Happy coding!

Prototype in JavaScript

Hey everyone. It’s been a few weeks since I last blogged. I’ve been on a vacation in Poland which was awesome! If you ever want to visit Poland I recommend visiting Gdańsk, a beautiful city, and Malbork Castle, the largest castle in the world!
Anyway, this blog isn’t about travel, it’s about programming, and prototype in JavaScript in particular!

So last time I finished my series on the MEAN stack, MongoDB, Express, AngularJS and Node.js. That meant a lot of JavaScript. Not just front-end, but also back-end. Next to my blog I’ve been writing a lot of JavaScript in my daily life too. And while writing all that JavaScript there was one thing I just didn’t quite understand, prototype. I’m not talking about the library with the same name, which kind of lost the battle for most popular all round JavaScript library to jQuery, I’m talking about prototypal object inheritance. I started asking around. I know some full-stack developers with years of experience in front- and back-end, but guess what? They didn’t fully understand it either! And then I went to look on the interwebs, but guess what? There just aren’t that many good prototype tutorials around. And that’s what this post is all about.

You can find the full code with examples on my GitHub in the repository for this post.

Prototype, you don’t really need it…

So when asking my friends why, after all these years, they still didn’t know prototype their answer was something along the lines of “I never needed it.” Sure, that was the reason I never looked into it before too. Truth is that you can write JavaScript apps and libraries without ever needing it. But if you want to write fast JavaScript code you better start using it!

So first of all, what is prototype and why is using it better than not using it? In JavaScript each function has a prototype property. Prototype has all the, I guess you could call it default, methods and properties that an object, created through that function using the ‘new’ keyword, should have. As we know any object in JavaScript can have any method or property, we can just define them at runtime as we go. And that’s probably the big difference between using it and not using it. With prototype you define methods design time, or up front.

Let’s look at an example. Let’s say I have a Person object, the constructor and usage could look as follows.

var Person = function (firstName, lastName) {
    var self = this;
    self.firstName = firstName;
    self.lastName = lastName;
    self.fullName = function () {
        return self.firstName + ' ' + self.lastName;
    };
};

var p = new Person('Sander', 'Rossel');
console.log(p.fullName());

So what happens when we call new Person(‘Sander’, ‘Rossel’)? First, two new instances of String are created, ‘Sander’ and ‘Rossel’.

Second, a new object is created. After that the object, representing a Person, gets three new properties, firstName and lastName, which are assigned the two strings, and the function fullName, for which a new function is instantiated. That last part is crucial, a new function instance is created every time you create a new person object. Now performance and memory wise this isn’t optimal. Let’s see how we can optimize this.

var Person = function (firstName, lastName) {
    var self = this;
    self.firstName = firstName;
    self.lastName = lastName;
};

var getFullName = function (person) {
    return person.firstName + ' ' + person.lastName;
};

var p = new Person('Sander', 'Rossel');
console.log(getFullName(p));

This time the function getFullName is created once and can be used for any instance of Person.

The benchmark

Don’t believe me? Let’s test that.

'use strict';

var benchmark = function(description, callback) {
    var start = new Date().getTime();
    for (var i = 0; i < 10000000; i++) {
        callback();
    }
    console.log(description + ' took: ' + (new Date().getTime() - start));
};

var Person = function (firstName, lastName) {
    var self = this;
    self.firstName = firstName;
    self.lastName = lastName;
    self.fullName = function () {
        return self.firstName + ' ' + self.lastName;
    };
};

var PersonNoFullName = function (firstName, lastName) {
    var self = this;
    self.firstName = firstName;
    self.lastName = lastName;
};

var getFullName = function (person) {
    return person.firstName + ' ' + person.lastName;
};

benchmark('Full name', function () {
    var p = new Person('Sander', 'Rossel');
    var n = p.fullName();
});

benchmark('No full name', function () {
    var p = new PersonNoFullName('Sander', 'Rossel');
    var n = getFullName(p);
});

That’s a bit of code, but what happens is that we create 10.000.000 (that’s ten million) instances of Person with the fullName function defined on the Person object and we create ten million instances of a Person without the fullName function and use the pre-defined getFullName function instead. Then we log the time both methods took.

It may surprise you, but the result I get differs greatly per browser. Here they are (times in milliseconds):

                IE      FF      Chrome
Full name:      3705    361     2805
No full name:   3121    21      222

What this shows is that Firefox is fastest by far and IE is, of course, really very slow, especially when it comes to the method without full name. All browsers are considerably faster (especially Chrome) using the getFullName method though.

Enter prototype

Now that we’ve seen that functions in your constructors aren’t really optimal, especially when they don’t use any internal state of an object, and that we’ve seen how to optimize that let’s look at a better way, prototype.

The use of an external function doesn’t feel right. Full name should be part of your Person object, but now we have to call some external function to get it. That’s where prototype comes in. Functions in JavaScript have a prototype property. Prototype defines static methods, like getFullName, and ‘pastes’ them on your objects through prototypal inheritance.

var PersonProto = function (firstName, lastName) {
    var self = this;
    self.firstName = firstName;
    self.lastName = lastName;
};

PersonProto.prototype.fullName = function () {
    return this.firstName + ' ' + this.lastName;
};

And that’s really all there is to it! We have now defined the fullName function on PersonProto’s prototype, which means every instance of PersonProto now gets the fullName function.

var p = new PersonProto('Sander', 'Rossel');
var n = p.fullName();

If we’d benchmark this we’d get about the same speed as we got earlier when using the getFullName function (really, try it).

To Prototype or not to Prototype…

You might be tempted to think it doesn’t matter whether you use a static method like getFullName or prototype, other than the manner in which you invoke the function (object.function() vs. function(object)). That isn’t true though, prototype has a few pros and cons compared to other methods. First of all, by using it, we’ve lost control over ‘this’. Earlier, when defining fullName in the constructor we could use ‘self’, which always points to the correct instance of Person. We’ve lost that benefit with prototype and ‘this’ is now dependent on the context in which the function was invoked. Below code will break.

var p = new PersonProto('Sander', 'Rossel');
console.log(p.fullName());
var fn = p.fullName;
console.log(fn());

While the following code will work just fine.

var p = new Person('Sander', 'Rossel');
console.log(p.fullName());
var fn = p.fullName;
console.log(fn());

It’s the same code on the outside, but fullName of Person points to ‘self’ while fullName of PersonProto points to this (which, in the second call, is ‘window’).

Another pro, maybe, is that prototypal functions cannot be deleted from an object (although they can be overwritten). The first bit will run fine and prints ‘Sander Rossel’, the second part will break.

var p = new PersonProto('Sander', 'Rossel');
delete p.fullName;
console.log(p.fullName());

// This will break.
// fullName is no longer defined after a delete.
var p = new Person('Sander', 'Rossel');
delete p.fullName;
console.log(p.fullName());

And remember that prototype objects are static, which means they’re shared by all instances of an object. You can overwrite the value per instance though.

PersonProto.prototype.friends = [];
var p1 = new PersonProto('Sander', 'Rossel');
var p2 = new PersonProto('Bill', 'Gates');
p1.friends.push(p2);
console.log(p1.friends[0].fullName() + ' is a friend of ' + p1.fullName());
console.log(p2.friends[0].fullName() + ' is a friend of ' + p2.fullName());

Guess what, because the friends array is static Bill Gates is now also a friend of himself, even though I only added him to my own friends.

Inheritance

One big pro to using prototype is inheritance. One object can inherit the prototype of another object. The ‘pseudoclassical’ way to do this is by copying the prototype object.

var Employee = function (firstName, lastName, salary) {
    var self = this;
    self.firstName = firstName;
    self.lastName = lastName;
    self.salary = salary;
};

Employee.prototype = PersonProto.prototype;

Employee.prototype.getSalary = function () {
    return this.fullName() + ' earns ' + this.salary;
};

So we have an Employee with a salary and, of course, a name. An Employee is actually just a Person with a salary. So we ‘inherit’ the prototype of PersonProto. We also define a new function on the prototype, getSalary. Unfortunately, while Employee now has the fullName function, this has some undesired side effects…

var e = new Employee('Sander', 'Rossel', 1000000); // I wish :-)
console.log(e.getSalary());

var p = new PersonProto('Sander', 'Rossel');
console.log(p.getSalary());

That’s right, PersonProto now also has a getSalary function! That’s because Employee.prototype == PersonProto.prototype. So once we add getSalary to Employee.prototype PersonProto now has it too.

We can fix this by creating a temporary constructor, assigning the prototype to the temporary constructor, instantiating an object using the temporary constructor and assigning that to the prototype of the inheriting object. Last, because a prototype is assigned to the constructor function, we must set the prototype’s constructor to the inheriting constructor.

var Temp = function () {};
Temp.prototype = PersonProto.prototype;
Employee.prototype = new Temp();
Employee.prototype.constructor = Employee;

Employee.prototype.getSalary = function () {
    return this.fullName() + ' earns ' + this.salary;
};

Alright, so I agree that’s pretty arcane! Luckily you can put this in a function so you only have to write it once.

var inherit = function (inheritor, inherited) {
    var Temp = function () {};
    Temp.prototype = inherited.prototype;
    inheritor.prototype = new Temp();
    inheritor.prototype.constructor = inheritor;
};
inherit(Employee, PersonProto);

And you can also see that Employee is actually also a PersonProto.

if (e instanceof Employee) {
    console.log('e is an instance of Employee.'); 
}

if (e instanceof PersonProto) {
    console.log('e is an instance of PersonProto.');
}

Object.create

Another method to create an object with a specific prototype is by using the Object.create function.

var o = Object.create(PersonProto.prototype);
o.firstName = 'Sander';
o.lastName = 'Rossel';
console.log(o.fullName());

So here o has the PersonProto prototype and o is of the type PersonProto. Now let’s use Object.create for inheritance like we’ve seen above (for this example I’ve created an Employee2 which is similar to Employee).

Employee2.prototype = Object.create(PersonProto.prototype);
Employee2.prototype.constructor = Employee2;

Employee2.prototype.getSalary = function () {
    return this.fullName() + ' earns ' + this.salary;
};

var e = new Employee2('Sander', 'Rossel', 1000000); // I wish :-)
console.log(e.getSalary());

And there you have it.

Prototype vs. __proto__

As I said before prototype kind of ‘glues’ the static functions to the object instances you create. This happens through the proto property each object instance has. Each instance of an object has a proto property defined, which is constructed using the constructor.prototype. So while object literals don’t have a prototype property they do have a proto property which you can use to dynamically extend the prototype of an object.

var p = {
    firstName: 'Sander',
    lastName: 'Rossel'
};

if (p.__proto__ === Object.prototype) {
    console.log('__proto__ points to the constructors prototype.');
}

// Highly discouraged!
p.__proto__.fullName = function () {
    return this.firstName + ' ' + this.lastName;
};

console.log(p.fullName());
delete p.fullName;
console.log(p.fullName());

So by adding fullName to o.proto we’ve actually added fullName to Object.prototype! That means ALL your objects now have a fullName function.

var i = 10;
console.log(i.fullName());

See why doing that is highly discouraged? It’s slow to boot. I just thought you should know about it.

Beware, fellow traveler!

So that’s it! I have some final remarks for you before you go and use this knowledge out in the wild.
Don’t change the prototype of built-in types such as Object and Array. You could do this to support new functionality in older browsers, but there are probably a few libraries that already do that for you.

Prototype can have a negative impact on performance. What happened when we called fullName on Employee? The JavaScript engine looks for fullName on the object, but could not find it. It then looked for fullName on the Employee prototype, but again couldn’t find it. It works its way down the prototype chain and looked in the PersonProto prototype next, where it found fullName and called it. You can imagine that having huge inheritance chains can be bad for performance (but why would you have huge chains anyway?).

Last, as I said before, you can use JavaScript without ever needing prototype. Likewise, you can inherit objects without the need for prototype. For example, you can create a function Employee that creates a Person, adds the salary property and the getSalary function, and returns that.

Wrap up

And here’s some additional reading. First, of course, we have JavaScript Succinctly, which explain the various types, objects and functions, including prototype and inheritance.
And if you have some money to spare I can recommend one of the following books: Object-Oriented JavaScript by Packt Publishing or The Principles Of Object-Oriented JavaScript by No Starch Press.

Happy coding!

MEAN web development #9: Some last remarks

So by now we’ve seen the full MEAN stack, MongoDB, Express, AngularJS and Node.js. Additionally we’ve seen the templating engine Jade in action as well as used Socket.io for real time web applications. If you combine that with my earlier series on ‘vanilla’ web development you’re already a pretty versatile web developer (well, dependent on how much you practiced)!
In case you missed a MEAN article, here they are:

  1. MEAN web development #1: MEAN, the what and why
  2. MEAN web development #2: Node.js in the back
  3. MEAN web development #3: More Node.js
  4. MEAN web development #4: All aboard the Node.js Express!
  5. MEAN web development #5: Jade and Express
  6. MEAN web development #6: AngularJS in the front
  7. MEAN web development #7: MongoDB and Mongoose
  8. MEAN web development #8: Sockets will rock your socks!
  9. MEAN web development #9: Some last remarks

In this post I’m going to discuss one topic we haven’t covered yet, deployment. Next to that I’m going to give you some alternatives for the technologies we’ve covered in this series.

There is no GitHub repository for this article as there won’t be any code samples.

Deploying a Node.js application

So we haven’t talked about deployment yet. Not even in my web development series. To be honest, deploying software is not what I do, I just write them. Besides, deploying to the Windows laptop you’ve used to build your software is a bit difficult and in most cases isn’t even close to actual deployment. Unfortunately I don’t have any spare web servers laying around 🙂
That isn’t to say I can’t point you in the right direction.

Node.js is a little different from what you’re used to. Consider the PHP application we wrote earlier in Web development #4: PHP in the back. We wrote our page and got it up and running using XAMPP, where we specified the port and other settings. The PHP file didn’t do anything. Likewise, a C# web application doesn’t do anything until you host it using IIS, which can be configured however you like.
That’s different for a Node.js application. After all we specify the server, where it should run and how it should run in our JavaScript file. And then we could run it from the console. On production environments we really don’t want to run a console though. We probably want to host our application on port 80, which is the default HTTP port, which requires an elevated command prompt. So someone with admin rights should log on to the server to (re)start our Node.js app every time something happens (either the app crashes or the machine is restarted). That doesn’t sound very appealing…

So we could simply write some command script using a loop that restarts Node.js whenever it crashes and start that up, with elevated privileges, either using a service or the built-in scheduler. That solution has some serious limitations though. What if your app goes in an unrecoverable state, making it crash in a loop? Or what if, for some reason, it hogs up all of your memory? A command prompt can’t really give you a detailed log of what’s happening with your application.

That sounds really tiresome and messed up. Luckily there is a better alternative. You can use PM2 (Process Manager 2) to run Node.js (PM2 on GitHub). PM2 can run other scripts such as PHP, CoffeeScript and Ruby as well and it works on Linux, MacOSx and Windows.
You can install PM2 using npm (npm install pm2 -g). Make sure to add  the -g to make a global install. Now you can simply start any Node.js app by running “pm2 start node_app.js”. “pm2 list” will show a list with running applications. Then you can generate a startup script with “pm2 startup” to automatically start your apps on a computer reboot (which unfortunately doesn’t work on Windows, so I haven’t been able to test it). “pm2 monit” will show you some statistics on the current status of your applications.
Next to that PM2 has a deployment system, log management, a development tool (similar to nodemon which we’ve been using in this series), cluster mode (for running on multiple CPU’s, remember Node.js is single threaded) and much more which you can all use as well.

There are some alternatives to PM2, such as forever, but something completely different is to run your Node.js app as a Windows service. You can do this by using NSSM (the Non-Sucking Service Manager) and winser. You can download NSSM from their website and you can install winser using npm (npm install winser). Then you should add the following lines to your package.json:

"scripts": {
    "postinstall": "winser -i -s -c",
    "preuninstall": "winser -r -x -s",
}

Now in the command prompt browse to the folder that contains the package.json and execute “node_modules.bin\winser -i”. You have now installed your Node.js application as a service. To uninstall execute “node_modules.bin\winser -r”.

So I know that’s not much, but at least these are some considerations when deploying a Node.js application.

HTTPS/SSL

You might want to deploy your application using HTTPS for secure SSL-encrypred traffic. For this you’re going to need some certificates, which I dont have. Node.js has an https module, just like it has a http module. So now it shouldn’t be too difficult to write yourself an HTTPS enabled Node.js server.

var https = require('https');
var fs = require('fs');

var options = {
    key: fs.readFileSync('some-key.pem'),
    cert: fs.readFileSync('some-certificate.pem')
};

var server = https.createServer(options, function (req, res) {
    // Stuff...
});
server.listen(80, '127.0.0.1');

And here’s how you can use Express to handle HTTPS.

var app = require('express')();
var https = require('https');
var fs = require('fs');

var options = {
 key: fs.readFileSync('some-key.pem'),
 cert: fs.readFileSync('some-certificate.pem')
};

// Stuff...

var server = https.createServer(options, app`);
server.listen(80, '127.0.0.1');

And now you also know why you should use packages for your routing, you may want to re-use them.

MEAN alternatives

In this series we’ve seen the MEAN stack: MongoDB, Express, AngularJS and Node.js. It offers an alternative to the traditional LAMP stack, Linux, Apache, MySQL and PHP and the Microsoft stack with .NET and IIS. The baseline here is that we programmers just like to come up with cool or clever sounding acronyms. As I said before you can have a perfectly good MEN stack, leaving out the AngularJS. Or put in some Jade, or Sockets.io. Well, people have actually done that and came up with some other nice sounding acronyms. Here are some that I’ve seen: ANNE (AngularJS, Node.js, Neo4j, Express); BEANS (Bootstrap, Express, AngularJS, Node.js, Sockets.io); EARN (Express, AngularJS, Redis, Node.js). My point is that you’re not tied to MEAN, it’s simply a couple of technologies that work well together and can help you get things done.

If you like you can have a look at Sails.js, which is built on Express, but adds MVC and an ORM, among other things. Another alternative for Express (or Sails.js) is Hapi.js, which focuses more on configuration than code.

As alternative for MongoDB we’ve already seen Neo4j (a NoSQL graph database) and Redis (a NoSQL key-value store). They don’t actually have to be alternatives as you can use both in a single project. An actual alternative could be CouchDB, which is also a NoSQL document database and is probably the second most popular document database after MongoDB,
But who said you have to stick to NoSQL? If you’d like to use MySQL with Node.js that’s perfectly fine too!
If you just look at the Express database capabilities you’ll find databases such as Cassandra, PostgreSQL and ElasticSearch are supported out of the box.
And there are drivers for SQL Server and Oracle too.

Let’s look at some AngularJS alternatives. Of course there’s always jQuery that you can still use in your projects. Get some Knockout.js in your project too and you might not need AngularJS at all (although AngularJS is probably your first choice for SPAs (Single Page Applications).
While AngularJS is the most popular front end framework, it’s not your only choice. Underscore.js is another popular framework that is a little less bloated than AngularJS (but does less as well).
Another framework that you might want to try is Ember.js, which does pretty much everything that AngularJS does too. If you want to read more about Ember.js I can recommend Erik Hanchett’s blog.

And if you wish for another HTML template engine (or no template engine at all) you can ignore Jade and go with, for example, Mustache or Handlebars. Keep in mind you can use templating both on your back end and in your front end.

I’d mention alternatives for Node.js, but that would kind of defeat the purpose of MEAN and the (almost) all JavaScript stack. So let’s not do that (besides, you probably know them already).

Other popular modules

Let’s quickly check out some other popular Node.js modules. I’ll let you figure out how to use them on your own, but I’ll give a quick overview and a link to the website (with documentation).

The first library you might want to check is Browserify, which let’s you use require() in the browser. That’s pretty sweet!

Another popular library that you can use in both Node.js and in your front end is async. It provides many functions for asynchronous execution, such as each, map and filter on arrays.
It can also help you with asynchronous flow control. That is executing one task after another in an elegant and asynchronous manner. And for this same purpose the creator of async also gives us Nimble. And you might want to take a look at the alternatives Seq and Step as well.

So how about your minification, compilation, unit testing, linting (checking for syntax errors) etc.? For these tasks you can use Gulp or Grunt. Both applications are popular ‘task runners’ and can be easily installed using npm (using -g) and automate these kinds of tasks. They might take some configuration and some time to get used to, but they’re worth it.

Last I’d like to point out commander. Since everything nowadays seems to be working with the command prompt you might want to make your application command prompt configurable too. Something like “node server.js -super” where -super (or -s for short) starts your application in superman mode. You don’t need a library for this, but it does come in handy. Commander seems to be the most popular one.

So that’s it for the entire MEAN series. I hope you’ve learned and enjoyed it as much as I have! This is the end of the MEAN series, but certainly not the end of my blogging career. In the future you may expect more posts on web development, JavaScript, NoSQL, and who knows what. I’m always open for suggestions!

For additional reading on MEAN and related technologies I can once again recommend the Succinctly series by Syncfusion. They have many free ebooks like Node.js Succinctly, AngularJS Succinctly and MongoDB Succinctly. I’m also a big fan of the books by Manning Publications. Though not free they’re definitely worth your money. Books like Node.js In Action (second edition coming up), AngularJS In Action, MongoDB In Action (also a second edition coming up) and even Express In Action. A book on the entire MEAN stack, Getting MEAN, is also in the writing (so get it early)!

I’ll be on a vacation in Poland for the next two weeks, so don’t expect a new blog post for at least another three weeks. Maybe a short one on a rainy day and if I have wi-fi in the hotel, but I’m not making any promises.

Thanks for sticking with me and happy coding!