Configure a Life Cycle Management in Azure Storage Account

 Azure Blob Storage lifecycle management offers a rule-based policy that you can use to transition blob data to the appropriate access tiers or to expire/cleanup data at the end of the data lifecycle.

Select Azure Storage account

Access Lifecycle Management

Add below rule to cleanup all the blob data older than 7 days.

 

You can use the list view also to configure the rule from UI. e.g.

• Lifecycle management policy definition Sample: https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview#lifecycle-management-policy-definition
• Lifecycle management rule definition Sample: https://learn.microsoft.com/en-us/azure/storage/blobs/lifecycle-management-overview#sample-rule

Thanks for reading, CloudOps ⌂Signing Off! 😊

 

Setup Azure function (ServiceBus and HTTP based trigger ) with Docker Container Image

 To setup Azure function we first need to choose the azure hosting plans.

• Consumption Plan
• Premium Plan
• Dedicated Plan (App Service Plan)

In this blog, we will see how to create a Azure function using App Service plan and configured it with Docker container image.

Once you have your azure function created in azure you can start configuring triggers.

• Create a file called function_app.py using VS code.
• Add below code to it.

import azure.functions as func

import json

import logging

import time

app = func.FunctionApp()

#Function 1 - ServiceBus Triggered

@app.function_name('ServiceBusTrigger-Function')

@app.service_bus_queue_trigger(arg_name="message",

                               queue_name="my-servicebus-queue-name",

                               connection="ServiceBusConnectionString")

def main(message: func.ServiceBusMessage):

    logging.info(json.loads(message.get_body()))

   

    logging.info("Start: The time of code execution begin is : %s", time.ctime())

    time.sleep(300)

    # your processing logic

   

    logging.info("End : %s", time.ctime())

#Function 2 - Http Triggered

@app.function_name('HttpTrigger-Function')

@app.route(route="payload", auth_level=func.AuthLevel.ANONYMOUS)

def get_metadata(httpReq: func.HttpRequest) -> httpReq.HttpResponse:

    return func.HttpResponse(json.dumps(httpReq.get_body()))

•  local.settings.json file to store function parameters value to run azure function locally.

{

  "IsEncrypted": false,

  "Values": {

    "AzureWebJobsStorage": "DefaultEndpointsProtocol=https;AccountName=stg;AccountKey=abcdef==;EndpointSuffix=core.windows.net",

    "ServiceBusConnectionString": "Endpoint=sb://my-sbns.servicebus.windows.net/;SharedAccessKeyName=SharedAccessKey=abcdefghi=",

    "FUNCTIONS_WORKER_RUNTIME": "python",

    "my-servicebus-queue-name": "queue-name"

  }

}

• We can use App Service - App Settings to configure the parameters values, Also we need to set Docker image related parameters for Deployment.

[

  {

    "name": "APPLICATIONINSIGHTS_CONNECTION_STRING",

    "value": "InstrumentationKey=abcdef-4tgh-4567-jkih-tryrtyrtyr;IngestionEndpoint=https://ai.azure.com/;LiveEndpoint=https://es2.livediagnostics.monitor.azure.com/",

    "slotSetting": false

  },

  {

    "name": "AzureWebJobsStorage",

    "value": "DefaultEndpointsProtocol=https;AccountName=acctest;AccountKey=abcde==;EndpointSuffix=core.windows.net",

    "slotSetting": false

  },

  {

    "name": "DOCKER_REGISTRY_SERVER_PASSWORD",

    "value": "abcdefghijklmnopqrstuvwxyz",

    "slotSetting": false

  },

  {

    "name": "DOCKER_REGISTRY_SERVER_URL",

    "value": "https://mydemoacrtest.azurecr.io",

    "slotSetting": false

  },

  {

    "name": "DOCKER_REGISTRY_SERVER_USERNAME",

    "value": "mydemoacrtest",

    "slotSetting": false

  },

  {

    "name": "FUNCTIONS_EXTENSION_VERSION",

    "value": "~4",

    "slotSetting": false

  },

  {

    "name": "FUNCTIONS_WORKER_RUNTIME",

    "value": "python",

    "slotSetting": false

  },

  {

    "name": "ServiceBusConnectionString",

    "value": "Endpoint=sb://my-sbns.servicebus.windows.net/;SharedAccessKeyName=;SharedAccessKey=abcde=",

    "slotSetting": false

  },

  {

    "name": "WEBSITES_ENABLE_APP_SERVICE_STORAGE",

    "value": "false",

    "slotSetting": false

  }

]

This is the folder structure.

Thanks for reading, CloudOps ⌂Signing Off! 😊

Azure Function ServiceBus Trigger is triggered repeatedly the same message after about 10 minutes from Azure Service Bus Queue

I have a Web Application hosted in App Service which sends Message to my Azure ServiceBus Queue - this works fine. Then I have my azure function running under an App Service Plan (Dedicated Plan), which is a ServiceBusTrigger based, receives message from Azure ServiceBus Queue. The problem is that after about 10 minutes, when the first (previous) trigger process has not completed its work, Azure Function receives message from Servicebus queue again. My azure function executing logic can take up to 50-60 minutes.

Initially I was thinking the cause can be in Azure function either it is getting timeout or somewhere it is not behaving properly but while investigating 2 days with simulation of longer duration 20 minutes, 30 minutes we came to know that this is due to ServiceBus default behaviour.

ServiceBus have MaxAutoRenewDuration Property which is responsible for maximum duration within which the lock will be renewed automatically for a particular message in servicebus queue, however there is relation of it with MaxDeliveryCount.

The maximum MaxAutoRenewDuration  for servicebus message is 5 minutes so if message is not processed by your azure function within 5 minutes(By default, the runtime calls Complete on the message if the function finishes successfully, or calls Abandon if the function fails.) it will increase the delivery count and re-try/re-queue the same process till delivery count reach to its limit. once delivery count limit reached, the servicebus queue message moved to DeadLetter Queue.

Not sure why message requeued in every 10 minutes if the maxautolock duration is just 5 minutes, what I noticed several times - it was always double the autorenewduration, if it is 5 minutes the function will retrigger or message get re-queued in 10 minutes, if it is 3 minutes, function will re-trigger in 6 minutes something that sort).

To resolve this requeue/re-triggers issue in every 10 minutes we need to update the AutoRenew duration. The maxAutoRenewDuration is configurable in Azure function host.json file , which maps to ServiceBusProcessor.MaxAutoLockRenewalDuration. The default value of this setting is 5 minutes.

https://learn.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus?tabs=isolated-process%2Cfunctionsv2%2Cextensionv2&pivots=programming-language-python#hostjson-settings

{

  "version": "2.0",

  "extensions": {

    "serviceBus": {

      "messageHandlerOptions": {

        "autoComplete": true,

        "maxConcurrentCalls": 16,

        "maxAutoRenewDuration": "00:50:00"

      }

    }

  },

  "functionTimeout": "-1",

  "logging": {

    "applicationInsights": {

      "samplingSettings": {

        "isEnabled": true,

        "excludedTypes": "Request"

      }

    },

    "logLevel": {

      "default": "Warning",

      "Function": "Warning",

      "Function.Accelerator.User": "Debug"

    }

  },

  "extensionBundle": {

    "id": "Microsoft.Azure.Functions.ExtensionBundle",

    "version": "[4.*, 5.0.0)"

  }

}

Thanks for reading, CloudOps ⌂Signing Off! 😊

Azure function Timeout after 10 minutes

I have my azure function running python app and recently I faced an issue about azure function being timeout after every 10 minutes if process taking more than 10 minutes. This is because of default timeout settings we have for each hosting plans for azure function.