Abstract

The Problem

The Solution (An Alternative)

The Proposition

How does this look like in the real world?

A Plausible Scenario: Describe a plausible scenario involving the gig economy but bounded within the corporate structure

Describe the interactions between business, broker and technical engineers/software developers

Describe the technologies facilitating the interactions between business and developers and the market

Core Principles

Software Development Teams cannot keep up with Market Demands for Advanced Technology: The speed of modern technological advancement and increasing market demand is not matched by the production systems used to deliver software and technology in general and software corporations no matter how agile seem to fall short of producing effective solutions to meet the market demand.

A major contributing factor to this inability is poor resource allocation: This inefficiency is due to the inability of these management systems (Agile, Scrum, Waterfall, Spiral and their practitioners - the businesses that run them) to allocate human and technological resources efficiently in response to market demands.

Yet, highly effective resource management technologies and frameworks exist - yet to be tried: The Free Market, represented by the Capitalist Model is widely acknowledged to be the undisputed most efficient means of resource allocation system in history, yet the principles of this model are not leveraged within organisations to achieve the same level of resource allocation efficiency.

We posit that this means should indeed be leveraged to yield exactly the benefits seen in the global market system.

The Market Place of Ideas is Not Fully Democratised with current Software Development Methodologies:

While the democratic nature of the Capitalist Free Market system is widely acknowledged to yield many benefits in terms of innovation and creative freedom as well as speed of production, this aspect is hardly every leveraged in modern corporate structures even in systems as flexible as Agile or the Spiral Method and certainly not in Waterfall.

We posit that this characteristic of the modern capitalist free market system should indeed be leveraged to yield exactly the benefits seen in the global market system. There is a mismatch between the capabilities of available technology and the ability of IT and Software Development Teams to Leverage it: The capabilities of the information technology of the modern era are far beyond the capabilities of software development teams and methodologies to leverage them effectively. For example, Cloud offers resource allocation flexibility and automation features which most IT companies are unable to exploit to the maximum. Before the cloud era, systems like VMS, SUN Solaris Cluster, AIX, IBM Power-series and other "Mainframe" technologies possessed advanced and highly granular resource management and automation capabilities which remain un-exploited to this day.

New Relationships:

The Business - Software Team Relationship,

The Business - Intermediary - Software Team Complex,

Relationship between Software Development Team Members,

Relationships between Competing Teams.

The Business

The Software Development Team

The "Brokers"/"Middlemen"

The "How": Enabling the Software Development Team Market Model

Enabling Technologies:

Dynamic Contracts,

Granular work metrics and value estimation, precision time and labor tracking. Bidding exchanges, stock value instead of KPIs.

Enabling Organisational StructuresEnabling Legal Structures: Dissolution of the Non-Compete/Exclusivity Law.

Strengths of the Model

From the perspective of the employee

From the perspective of the employer

From the perspective of the middleman/broker

From the perspective of the SW Developer Team

From the perspective of The Market (Customer)

Weaknesses

Highly Competitive (Potentially Darwinian)

Subject to volatile market forces

New Opportunities Enabled by the Model

The Death of Venture Capitalists: Dynamically Emerging and Collapsing Ventures

The Democratization of Venture Capital (GoFundMe)
Potential Risks

Vulnerable Employees will suffer from market volatility

Vulnerable Enterprises will suffer from market volatility
Long Term Evolution of the Model: The Evolutionary Horizon: Dystopia or Utopia?

The Power Relationship between Employee and EmployerCustom Knowledge vs. Brute Artificial Intelligence

The Future of Intellectual Property

Robotics and the Role of Roboticization

Managing Change and Countering Resistance

There is Room for all:

Traditional Enterprises will remain "as is"

The Model will prove itself or perish

Conclusion

Such a model is certainly possible and plausible given the current trends in market and technology

But will require specialized software and systems to enable a true market - like unto a stock exchange

Potentially empowering to both employers and employees within the context of the Gig economy

While impacting the traditional Power relationship between employee and employer

With implications for IP via the concept of "Custom Knowledge" unique to the individual.

Having the potential to impact society in both dystopian and utopian ways.

In fact, there should be a threshold of availability and stability before which chaos engineering should not even be considered.

Otherwise it becomes a waste of engineer time and business resources.

In the modern era cloud infrastructure is often deployed using declarative code like Terraform or Bicep or less declartively using Cloud Development SDKs e.g "Pulumi" (TM).

This includes network topologies built from cloud IaaS components like VNETs (with private IP address space), Virtual Network Gateways, Cloud-to-Datacenter links (e.g ExpressRoute), Virtual Network peering links, "service endpoints" and Public facing IP addresses to communicate with services on the internet.

For large organisations (including "Enterprises") the management of private address space is complex enough "on premises", let alone in the cloud.

Usually a dedicated networking team has to be involved in assignment, provisioning, reclamation, renumbering of IP address space to cloud teams.

This task is often carried out by a dedicated networking team using manual or semi-manual tools ranging from Excel Spreadsheets to an enterprise product like InfoBlox (TM) IPAM.

The management of DNS records is often related to this process if a centralised tool like InfoBlox is in place.

Cloud Engineers then consume the networking information managed with these tools to configure IaC for the cloud using a separate workflow.

Problem Statement

The management of IP address space in the cloud is an ongoing, often manual, iterative task involving mistakes and a degree of trial and error in getting address space correctly subnetted, assigned, configured.

When this process is required to configure infrastructure code for the cloud (e.g Terraform IaC) it can result in code errors and deployment failures due to increasing complexity of configuring the code to reflect the right IP address assignments.

All this would be manageable if the infrastructure code (often written in terraform or bicep), automation (CI/CD pipelines) infrastructure "state" were robust and recovered from configuration errors easily.

However, this is emphatically not the case in general for the cloud. The following problems are common:

• Networking configuration errors lead to IaC deployment failures at deployment time in the cloud even if successfully passing tests prior to actual cloud deployment, simply because of hidden business rules  of the Cloud Platform Vendor.
• Recovery from those errors loses hours or days of DevOps team manpower.
• In a complex organisation with an extensive cloud hub-and-spoke network topology the errors can add up to days of downtime in a given year resulting in service downtime and frustrated DevOps engineers.
• To sum up: The more complex an organisation's networking requirements are in the cloud the greater the need for a reliable system of managing IP address pools in code.

What is the ask?

As a Network Manager responsible for managing address space across multiple clouds, I want to be able to provision a pool of address space to VNETs, subnets, interfaces, physical links, gateways on the cloud flexibly and track the utilisation by teams and projects and ownership of address space.

I want primarily to:

• Provision and reclaim address space as applications are deployed and removed from the cloud,
• Track the history of ownership
• Make IP address space available to Infrastructure Code via automated API queries.

In addition to these core requirements (or "wants"):

• I want smooth integration of IP address Provisioning with CI/CD pipelines and network automation tools without having to manually do subnet calculations every time I assign a range from the reserved pool of available IP addresses.
• I would prefer Cloud Application or Engineering teams manage the detailed subnet allocations in the cloud for their specific applications directly rather than relying on the Network Team.

What I want to avoid:

• I do not want to maintain IP address records using a system of documentation based on MS Excel or notepad on the specifics of address assignment in the cloud. While these documents should be available on demand, they should not be the core means of managing IP address space in the organisation.
• I do not want to buy a million dollar Infoblocks solution to manage my entire organisational address space - I have enough infrastructure to maintain as it is.

What Problem Does This solve?

Reduction of Administrative Errors:

Management of IP addresses in a consistent, reliable and error free way prevents automation failures when provisioning IP addresses to cloud infrastructure due to address space overlap.  Moreover, it reduces the cascading Infrastructure Code bugs and failures which result in application downtime when mistakes with IP address assignment inevitably happen.

Improvement of configuration consistency:

The IPAM solution would also carry out tasks like ensure IP address ranges remain contiguous as they are allocated/de-allocated in a VNET/VPC.

Smoother, Faster Cloud Deployment Workflow:

Automating the provisioning and assignment of address space as well as the consistent management of addresses speeds up the cloud deployment workflow and reduces the I.T workload on teams.

What Value Does This Add?

Increased reliability in IP address management leads to more resilient Cloud automation and infrastructure overall, which in turn saves costs and time spent maintaining infrastructure. Unifiying the IP address management workflow and the cloud configuration workflow speeds up operational processes and delivery dramatically (if done right!).

This reliability and operational efficiency can be achieved with effective automation (note the emphasis!).

The advantages for automation (including self service) seem compelling in terms of opportunities for operational agility:

• Networking and Infrastructure Teams could pre-provision broad ranges of address space in the IP address database  for the Cloud following which Cloud Engineers could allocate or de-allocate ranges "programmatically" to teams, functions, applications, other organisations.
• Application teams deploying infrastructure to our cloud could access IP address provisioning information to configure their Infrastructure Code in the deployment pipeline at run time, or prior. They could access the correct IP address ranges for any given application programmatically without needing the help of Cloud Infrastructure Engineers. All of this could be part of a single unified automation workflow instead of separate processes requiring manual coordination between teams.

Build Tasks: A quick brainstorm

How would we build this "Cloud Native IPAM Solution" ?

Because we're "agile" we jump straight into thinking about what we should build. After all, who in this era has time for "architecture" right???

The following key components spring to mind at first glance:

• IPAM IP calculation engine (IP address calculator)
• IPAM query and update interface (API, CLI and/or U.I)
• IP Address space tracking database of some sort
• Provisioning/Deprovisioning workflow along the lines of a CI/CD pipeline
• Integration with IaC through templating engine

At a very high level, our "Cloud Native IPAM Solution" fits into the enterprise as pictured below:

A little Architecture Wouldn't Hurt, Would it?

Well, maybe we shouldn't dismiss "Architecture" so hastily, after all this thing is beginning to look a little complicated!

Let's draw some pictures!

At a slightly more detailed systems level, we envision the components of the IPAM solution to have the following interrelationships:

The solution can be described as follows by enumerating it's components (in summary):

1. Data Model: Provides an abstract scheme for collecting all the properties of IP addresses and address spaces and their interrelationships as one consistent information structure.
2. Business Logic Engine:  Program logic which applies the correct rules for operating on the Data Model for IP addressing. This could be code which provides operations for manipulating the information in the data model according to allowed rules.
3. Relational Schema: The SQL relational schema implementing the data model in a format which can be represented in a database.
4. Relational DB: An actual physical database which hosts the relational schema of the data model and provides features for managing the information and the model programmatically.
5. Database Storage: The physical storage medium which will be used by the relational database software to store the actual ip addressing data.
6. API: An interface which allows programmatic access to the features of the data model and the information within the data model. This would include commands to allocate ip address ranges, search for available addresses, decommission address space/ranges, assign to applications and teams and check the status of addresses. The API would be consumed by users and pther scripts via web interfaces, CLIs and internet protocols like REST, Websockets, gRPC.
7. Deployment Automation Process: An automated process (e.g script, ci/cd pipeline, a.i agent, management service) which consumes IP address information to configure software running in The Cloud or generate configuration code like "Terraform" and "Bicep" (IaC DSL).
8. User Portal: The User Portal provides a very lightweight interface to inspect the IP address information in the IPAM database, annotate IP address information and carry out CRUD operations on it.

Getting Down to Nuts and Bolts

After this rather "lean" design exercise, we should be ready to translate our moderately abstract description of the solution into actual technologies and components which could be used to built a prototype.

To do this we transform our eight-point description into a more detailed specification of the system:

1. Data Model: Provides an abstract scheme for collecting all the properties of IP addresses and address spaces and their interrelationships as one consistent information structure.
2. Business Logic Engine:  Program logic which applies the correct rules for operating on the Data Model for IP addressing. This could be python script code which provides definitions wrapping operations for carrying out subnetting calculations, update of address metadata (CRUD).
3. Relational Schema: A SQL relational schema implementing the data model in a format which can be represented in a database.
4. Relational DB: An actual physical database which hosts the relational schema of the data model and provides features for managing the information and the model programmatically.We should prefer a "service-less" database to reduce the number of moving parts in the solution perhaps SQLite, but if needed PostGreSQL.
5. Database Storage: The physical storage medium which will be used by the relational database software to store the actual ip addressing data. The type of storage medium (spinning rust, solid state/RAM) would be influenced by the degree to which we need the solution to scale.
6. API: An interface which allows programmatic access to the features of the data model and the information within the data model. This would include commands to allocate ip address ranges, search for available addresses, decommission address space/ranges, assign to applications and teams and check the status of addresses. The API would be consumed by users and pther scripts via web interfaces, CLIs and internet protocols like REST, Websockets, gRPC.
7. Deployment Automation Process: Here we'd want to build IP address configuration to be used in Terraform IaC for the deployment of user VNETs/VPCs and the subnet layout in them.
8. User Portal: The User Portal provides a very lightweight interface to inspect the IP address information in the IPAM database, annotate IP address information and carry out CRUD operations on it. This could be implemented using a containerized web frontend like Flask/ Django etc ...

Now Let Us Build!

We have enough of a picture of our IPAM solution in mind by now to implement a prototype/PoC and test if it integrates with Terraform infrastructure Code for our Azure Cloud Network Implementation.

The following tasks (in no particular order) will be the key steps in our build process:

• We will structure our Terraform code to consume IP address configuration provided by an external IP db, perhaps via environment variables in a deployment pipeline (Jenkins?) or by direct query to the IPAM API.
• We will implement a CI/CD pipeline to extract the IP addresses and metadata from IPAM via an API and configure it in Terraform IaC for specific deployments.
• We will implement a data model, translate it to a SQL schema, configure it in an RDBMS.
• We will implement a "rules engine" with an API to operate on the data in the db according to the data model.
• We will implement a lightweight API to allow programmatic access to the functions in the data model from external scripts and systems.
• We will implement a lightweight web U.I to carry out all operations on the data in the data model to maintain the IP address data.

But first: Principles

Hold on. Not so fast.

We should settle on architectural guiding principles to evaluate and ensure the success of our implementation, define expectations of the solution and build efficiently:

• Lean Architecture: Design the MVP first
• Scalability not a concern at this point
• Agile Implementation: Deliver a working solution quickly, simply.
• Evaluate based on fitness for purpose in an actual usage scenario
• Robust implementation: Avoid fragile components and design even at the cost of features.
• Containerize and package the entire self-contained solution in a highly modular fashion.
• COTS technologies with minimal code
• Monolithic design in the MVP to aid "self contained packaging"

Understanding the approach, we can now move on to the actual implementation ...

The Implementation

We begin with the Data Model, without which the design is essentially meaningless.

In brief: Conway's law states that the software and systems design an organisation produces tends to resemble, in architecture, it's organisational structure.

After two decades of digital transformation projects it's been my personal experience that Conway's law becomes especially visible during the process of migrating an organisation to a new technological paradigm or to an improved version of it's current paradigm.

Most importantly, without awareness of conway's law during the digital transformation process one can fall afoul of it's implications they are ignored in one's planning.

....

The customer requires a simple API-based service for a) shortening URLs (or otherwise encoding them) and b) retrieving the encoded/shortened URLs for lookup via a client browser (or bulk lookup tool ...).

(Don't ask me why such a thing would be useful - bit.ly springs to mind)

Let's begin with the high level architectural design and gradually drill down to implementation in code.

System Description

The System implements the following functionality:

• Accepts an HTTP/S POST  to shorten (encode) a URL from a browser client to an API endpoint.
• The API accesses functions to encodes the URL, stores it in a Redis database key-value table for later lookup
• A Separate API service accepts an HTTP/S GET request for the shortened URL, look-up the shortened URL in the Redis table and returns the decoded URL in an HTTP/S 302 REDIRECT response to the client browser. This allows the user client to access the decoded URL via http/s

The System design achieves the following non-functional requirements in the described way:

Architectural Diagram

Continuous Delivery Automation

To achieve Continuous Delivery or Continuous Deployment the following deployment pipeline is included in the design:
A Jenkins pipeline is defined and implemented using Jenkins declarative pipeline script to deploy the Infrastructure of the system as Code (IAC):

a) Terraform is used to create the AKS cluster, Redis cache service, Azure Load Balancer, DNS CNAME records. This code is versioned in a git repository
b) The Kubernetes Pod and ingress configuration is defined as Helm charts also stored in a separate Git Repo
c) Credentials and Secrets required by the service for accessing the Redis DB and other secure services are stored in a credential vault accessed by Jenkins pipeline to embed in the deployment. This ensures no secrets are stored in the git repos.
d) The Jenkins deployment pipeline itself is defined using either Jenkins Pipelinescript  or Groovy directly and kept as versioned code in Git.
e) "Continuous Integration" is implemented to improve quality of the deployment code by triggering a build of a development/staging system every time changes are made to a git master branch.
f) The Redis DB Schema is versioned git as code and deployed as a separate stage of the pipeline

Redis Database Table Schema

Shortened URL code field name
Actual URL Code field name
shortURL
trueURL

URL Shortening Mechanism

The URL shortening method is to run the URL through a HMAC sha1 hashing function with a random salt to increase chances of uniqueness.

Here we use HMAC-SHA1 because it is a one-way hashing function and does not allow the url to be decoded from the hash itself (you need to look it up in the Redis table)

$hash = hmac_sha1($random_salt , $my_url);

Core Service Components

1. The API Servers

• The two API servers are implemented as HTTP Servers in Golang or Python
• These handle PUT and GET requests separately
• The services are running in docker containers with separate pods
• An API server receives the PUT request to encode a URL and looks up the URL
• If URL is found in the Redis table, the API returns: "Duplicate" in response
• If URL is not in the Redis table, it encodes it using a simple shortening algorithm
• returns HTTP 200 OK with the short URL only on successful write to the Redis table
• An API server receives a GET request including the shortened URL and does a lookup
• if the short url is found in the table, the original URL is returned using a query
• This URL is formatted into a 302 redirect response by the API HTTP Service