The Purpose of Technology

Technology serves as a formidable enabler for organizations; it is powerful yet often expensive, intricate, and time-consuming. Generally, I categorize technology into four main areas of focus:

1. Storage
2. Compute
3. Information Generation
4. Scheduling & Orchestration

To excel in these four focus areas, technology organizations must develop several key capabilities:

• Infrastructure Engineering (both cloud and physical)
• Data Engineering
• Analytics Engineering & Data Science
• Software Development Engineering
• Technology Product Management (TPM)

The interdependence between these five capabilities and the four focus areas is significant:

• Infrastructure Engineering supports all areas by constructing the necessary framework (for instance, using AuroraDB and Redshift for storage, Lambda/EC2 for computation, and AirFlow for scheduling).
• Data Engineering facilitates storage, information generation, and scheduling & orchestration.
• Analytics Engineering & Data Science drive information generation.
• Software Development Engineering enables computation and information generation.
• Technology Product Management aligns all capabilities to address problems effectively and timely.

From these areas and capabilities, certain essential systems must be established within a technology organization:

• Comprehensive documentation (both in terms of infrastructure and the documentation itself)
• A ticketing system
• An asset tracking system
• A robust project management framework
• A code repository and deployment system
• An identity management system
• A system for managing access credentials and secrets

For this article, our primary focus will be on the five capabilities and the four areas of concentration; we will cover the necessary systems in a subsequent piece.

Organizational Structure

The most straightforward approach to organizing the technology team is to structure it around the five capabilities. This means appointing a senior leader for each capability (e.g., Director of X) who reports directly to the CTO. I advise against separating analytics from data science; my experience suggests that treating them as distinct often results in data science becoming an expensive endeavor disconnected from business realities. To mitigate this, I advocate for a senior Analytics leader to ensure alignment with business needs.

Unless your team of TPMs is substantial, I recommend they report directly to the CTO until there is a need for a Director of Technology Project Management. Embedding TPMs within functional teams can lead to conflicts of interest, where they prioritize the function's needs over the project's requirements. By maintaining their own vertical, TPMs can enjoy better career development opportunities.

Leadership Principles

Once you have established a functional organizational framework, the next critical challenge is how to lead the team effectively; in other words, the leadership tenets. There are countless perspectives on this, but I will share mine, which has been influenced by Amazon's successful, albeit imperfect, practices:

1. Always prioritize business needs; no technology initiative should be undertaken without understanding its impact on business progress.
2. Delegate responsibility; avoid micromanaging at all costs.
3. Clearly define ownership and ensure everyone understands their responsibilities and dependencies.
4. Foster trust while implementing verification processes that encourage team members to critically evaluate their assumptions.
5. Embrace honest disagreement, robust debate, and commitment to decisions; prioritize productive discourse over social harmony.
6. Appoint Single Threaded Leaders (STLs) for critical projects; part-time involvement yields subpar results.
7. Insist on thorough documentation for every project; no project should go live without adequate documentation.
8. Adopt agile development methodologies, accepting the associated risks and uncertainties; balance speed with high standards thoughtfully.
9. Construct components rather than monolithic systems to enhance scalability.
10. Encourage continuous development, avoiding "big bang" solutions for greater agility and ongoing improvement in technology products.
11. Empower all individuals, regardless of their level, to take necessary actions to drive projects forward; seniority should equate to increased responsibility for supporting others.

Each of these principles warrants its own deep discussion.

Common Challenges

The most frequent mistake is to disregard these leadership principles; they are rooted in lessons learned from past failures and successes. While all tenets are significant, three are particularly crucial: 1) prioritize business needs, 2) avoid micromanagement, and 3) promote honest disagreement and debate. If you focus solely on these three, you will naturally gravitate towards the remaining eight by engaging with those who are most knowledgeable about the work and by candidly addressing mistakes and failures. Adhering to just these three tenets will inherently lead you to develop a high-performing team.

However, I recommend utilizing the insights that inspired the other eight tenets to expedite your journey with less friction.

Another common issue is establishing an organization with excessive hierarchical levels and middle management. In my view, a flatter organization promotes quicker and higher-quality outputs. Growth, complexity, and project volume may necessitate deeper structures, but proceed with caution and skepticism. Middle management often exists primarily to sustain itself by creating low-value tasks that justify its existence. This can result in decreased productivity, costly technology solutions that fail to deliver value, and a diminished competitive edge as the focus shifts inward rather than on customer needs.

For those interested in further exploring high-performing technology organizations, please follow me here on Medium. Your engagement encourages me to write more on these topics. I will delve deeper into the leadership principles and other essential features in future articles.