Airplane Tasks in Python

Airplane Tasks in Python

August 19, 2023

AIRPLANE HAS BEEN DEPRECATED

With the recent news about Airplane being acquired by Airtable and shutting down March 1, 2024, this post can be considered archived and void of meaning.

Something like Windmill can used as a replacement for Airplane. Other alternatives like Retool, Prefect, ToolJet, and more exist as well.

If you’re interested in the disappearance of Airplane, this post by one of the ex-Engineers is fairly illuminating.

This post will be a fairly brief musing about common patterns I employ when writing Airplane Tasks to address internal tooling requirements. Airplane is a feature-rich internal tooling and automation platform. Tooling and automation exists in the form of Tasks or Views, though this post will only cover the former. Tasks can be written in TypeScript/JavaScript or Python and this post will focus on Python.

For starters, each Python Airplane Task relies on an airplane.yaml file that outlines the version of Python to use, the flavor of Docker image, the CPU architecture to build the Tasks for, and environment variables to use for the Tasks (if wanting all Tasks to inherit these environment variables):

python:
  version: "3.11"
  buildPlatforms:
    - linux/amd64

linux/arm64 can be used instead of linux/amd64 if deploying Tasks to self-hosted agents on the arm64 flavor of Fargate

Environment variables can also be specificed in the airplane.yaml file:

python:
  ...
  envVars:
    ECS_TASK_ROLE:
      value: <custom IAM role ARN>

In this example, a custom ECS_TASK_ROLE is specified which each Task will automatically use when executing. This can be extremely useful when this role has permissions that Airplane’s default run role does not.

When writing Tasks, the Task filename must be suffixed with _airplane, so for example:

your_task_airplane.py

This naming convention allows Airplane to actually register the Task in the UI, so both this naming convention and the @airplane.task decorator are required to make Task code usable in the UI. While the Airplane docs typically provide function-based examples, I prefer creating classes to encapsulate the Task logic rather than using a function. For example, instead of:

@airplane.task(
    name="Your Task",
    slug="your_task",
    description="An example Airplane Task written in Python",
)
def your_task(a_parameter: str=""):
    # core logic

this pattern instead offers a lot of flexibility at the expense of only a few more lines of code:

@dataclass
class YourTask:
    a_parameter: str = ""

    def run(self):
        # core logic

@airplane.task(
    name="Your Task",
    slug="your_slug",
    description="An example Airplane Task written in Python",
)
def your_task(a_parameter: str=""):
    return YourTask(**locals()).run()

While this is much the same on the surface, using a dataclass allows for the decorated function to pass in the Task parameters to the class leaving the class to handle the rest. Most of the time, a class will contain several methods and attributes that are either tied to the Task parameters or generated after its methods run. Handling the outputs and inputs of these methods is a lot more convenient when assigned to attributes instead of returning values from functions and assigning their output to variables that are then used as inputs to other functions.

For instance, here’s a contrived example to illustrate the usage of attributes:

@dataclass
class YourTask:
    # Task parameters
    service_name: str = ""

    # Generated via methods
    client: boto3.client = None
    buckets: Dict[Any, Any] = field(default_factory=dict)

    def get_client(self):
        self.client = boto3.client(self.service_name, ...)
    
    def list_buckets(self):
        self.buckets = self.client.list_buckets()

    def run(self):
        self.get_client()
        if self.service_name == "s3":
            return self.list_buckets()
        return {}

@airplane.task(
    name="Your Task",
    slug="your_slug",
    description="An example Airplane Task written in Python",
)
def your_task(service_name: str =""):
    return YourTask(**locals()).run()

Using class attributes allows for computed values to be easily assigned and used within the class.

In this example, the Airplane Task would take an optional service_name Parameter and initialize a boto3 client using the service name in the get_client method. That client would then be used directly in the list_buckets method to retrieve a list of S3 buckets if the service_name parameter matches "s3". Since the run method calls each of the class methods, it is the only method that needs to be called by the decorated function to run the entirely of the Task’s logic.

Obviously this is a simple example, but Tasks with many methods calling airplane.execute to directly call other Airplane Tasks or even calling external .run methods of those Tasks are possible. Given the flexibility of Python, the implementation details are really only limited by imagination and creativity once up to speed with how Airplane operates. It is not uncommon for several modules in a repository to be called within the same Task. So long as the imports are valid, they can all be incorporated within the Task’s class and be consumed without issue.

In my mind, the class is the Task itself and the decorated function is the interface for the Task. Passing in additional parameters, decorator arguments (so long as they’re valid), or anything else is also possible. I have written fifty or sixty separate Tasks that all share this pattern and it offers a great foundation to writing Tasks quickly. When creating a new Task, I know exactly how the Task code will be laid out and the only thing I have to think about is writing the core functionality of that specific Task.

Hopefully this quick post was illuminating. If you haven’t used Airplane before, I would definitely recommend that you try it out!