When the leaves start to fall in Central Park, New Yorkers brace for a season full of important traditions. Families clamor for a good view of the Macy’s Thanksgiving Day Parade, students get decked out for the big college games and locals get out of town to find the best nearby pumpkin patches.
It’s also an important time of year for the movers and shakers of the local tech community. Their developers, programmers and engineers can take advantage of the many annual conferences and summits that occur this time of year. Women Impact Tech kicks off on Nov. 2 followed by CyberTech NYC later in the month.
These meetups provide the perfect opportunity for an ambitious tech pro to catch up with leaders from top NYC companies like those we’ve featured in this month’s roundup. Many like SeatGeek have had to address scalability issues in recent years. The mobile ticketing platform is now set to support more than $500 million in generated revenue by the end of this year. What started just over a decade ago as a simple aggregator is now a powerhouse brand. SeatGeek has 33 million app downloads and is the official resale ticket partner for the MLB.
Keep your eye out for more great things from all of these companies as we head into autumn and push towards the end of 2023.
Trumid is a fintech company working to increase efficiency in credit trading with data and technology.
Describe what scalability means to you. Why is scalability important for the technology you're building?
Scalability, in my perspective as a CTO, is the lifeblood of growth. It's not merely the ability of our systems to meet rising business and market demands efficiently. It extends to scaling our entire technology organization. A crucial metric for us is how quickly new developers can contribute to our codebase. Beyond being just a technical term, scalability encompasses a spectrum of principles and practices that drive growth, refine communication and shape agile and adaptive systems for evolving demands. At Trumid, our aim is to streamline processes and communication so we can eliminate redundancy and advance toward our objectives swiftly. Effective communication is critical in ensuring each interaction is purposeful.
The end game? To offer an unmatched user experience where our technology seamlessly operates in the backdrop, ensuring the focus remains on the immense value and unparalleled client experience we deliver.
How do you build this tech with scalability in mind?
From a technology perspective, we aim for what we term “elastic scalability.” Beyond just machines and code, it’s about sculpting a flexible ecosystem. We aspire for our tools and technologies to adjust gracefully to varied environments, believing such automation can offer a streamlined and cost-friendly approach to evolving demands. Our vision for the technology we're developing is a platform responsive to the rhythm of our users’ necessities.
Our vision for the technology we’re developing is a platform responsive to the rhythm of our users’ necessities.”
Equally significant is organizational scalability. As we expand, I’m continually motivated by the goal of retaining the clarity of communication we had in our earlier, smaller days. It’s about protecting every team member’s agility and adaptability, not just in their present role but in their potential to grow and add value as we evolve.
Thus, when I talk about technology and scalability, it’s about envisioning a lively, adaptable tech realm and a cohesive team that resonates with our mission.
What tools or technologies does your team use to support scalability, and why?
Our approach to technology is rooted in adopting the leading edge rather than the bleeding edge. We’re early adopters of tools and technologies like Scala, Docker, Cloud, Kubernetes and Karpenter. Our mantra is “Everything as Code.” For us, every manual task beckons innovation and automation. Leveraging technology, we continuously seek methods to achieve greater efficiency with fewer resources.
Axio is a cyber risk performance management and planning tool that helps clients make informed decisions about cybersecurity.
Describe what scalability means to you. Why is scalability important for the technology you’re building?
Scalability encompasses a multifaceted approach. At its core, it’s about designing a system to sustainably accommodate an increasing number of users. Beyond this foundational aspect, scalability also means using tools and processes that empower a growing number of engineers to actively contribute to the platform. It’s about delivering intuitive functionality while ensuring that customers can benefit from it without the need for professional training or support. Scalability is crucial for the technology I’m building because it ensures not only robust performance under expanding demand but also adaptability and inclusivity as we allow both developers and end-users to efficiently engage and grow within our platform.
Scalability is crucial for the technology I’m building because it ensures not only robust performance under expanding demand but also adaptability and inclusivity.”
How do you build this tech with scalability in mind?
When constructing technology with scalability at the forefront, our approach is supported by several key questions during the requirements phase. We ask if the functionality can serve more than the one client requesting it. We also ask whether we can reuse the functionality or capitalize on existing components in our system to deliver it.
In today’s tech landscape, common scalable design patterns rely on small discrete functions that can be completed quickly. These functions are designed to avoid a long-duration constraint, accommodate simultaneous users and generally operate without retaining a persistent state. This approach allows us to deliver scalable services that are largely fungible and can be leveraged where they are needed most.
What tools or technologies does your team use to support scalability, and why?
Our team leverages a microservices architecture paired with containerization techniques, primarily with Docker and Kubernetes. We thrive on crafting flexible and reusable code, while our database management ensures swift modifications without necessitating production releases. Furthermore, we leverage mature and well-supported languages and technologies such as Java and React. By adhering to industry-recognized design patterns and best practices, and by uniformly applying technologies across our engineering department, we ensure a cohesive and scalable infrastructure as we grow from thousands to hundreds of thousands of users.
SeatGeek is a mobile ticketing platform that allows fans to buy and sell tickets to live events.
Describe what scalability means to you. Why is scalability important for the technology you’re building?
At SeatGeek, scalability is one of our most important and interesting challenges. When a fan walks up to a stadium on gameday or gets ready for an onsale for their favorite artist, we want them to have a delightfully simple and stress-free experience. Fans should be able to focus entirely on their live event experience without worrying about the ticketing process.
Ticketing, particularly onsales, is a fascinating engineering challenge because every ticket is a different product. An onsale can involve matching hundreds of thousands of fans to hundreds of thousands of products, all with different views, amenities, quantities and price points, in just tens of minutes. This poses unique technical, product and design challenges that we are constantly working on solving here at SeatGeek.
How do you build this tech with scalability in mind?
Addressing scalability is a key part of our process for new development. By publishing our technical plans and encouraging discussion across the R&D organization, we can take advantage of our wealth of engineering experience to tackle problems in scalable ways. Understanding what level of scale we are targeting for a given system is also an essential part of this process.
For existing systems, we look at performance under real-world conditions and in regular performance stress tests. Teams define objectives around latency and error rates to ensure we are not hitting scalability limits in steady-state. We regularly load-test our site with simulated traffic for gameday and onsale scenarios. Given our growth and how dynamic ticketing traffic can be, this is the most reliable way to find the limits of various components of our system. It then provides a “laboratory” to test the impact of code, caching or infrastructure changes.
What tools or technologies does your team use to support scalability, and why?
High-quality observability and load-testing capabilities are the most critical tools for understanding and improving scalability. We leverage Datadog extensively to understand how our system is performing and, importantly, why it is performing the way it is.
High-quality observability and load-testing capabilities are the most critical tools for understanding and improving scalability.”
We use Blazemeter to generate load according to our test plans. However, we have developed an in-house orchestration system for standing up full production-scale environments, getting test fixture data in place, executing multiple performance test scenarios and then tearing down the environment and reporting results. This allows us to consistently simulate high-traffic scenarios at a higher frequency than we see in the real world.
Truehold is a real estate and home services company specializing in residential sale-leasebacks.
Describe what scalability means to you. Why is scalability important for the technology you’re building?
Scalability means making each customer feel like they’re the most important customer. At Truehold, we bridge multiple tools, each designed around processes to solve one part of the puzzle. We then bring the key outputs to the customer in the form of our sale-leaseback offer. Some of these tools, and some of the processes that follow, aren’t relevant to the customer. That means we work to minimize the unrelated aspects and embrace elements of eventual consistency.
How do you build this tech with scalability in mind?
Each time we tackle a new problem, we ask ourselves the same set of questions about the ideal user experience and how we can respond to the customer in the best manner. We also consider how much time we have, how important is this, to whom it is important and what importance it will have on our environment.
Each time we tackle a new problem, we ask ourselves the same set of questions about the ideal user experience and how we can respond to the customer in the best manner.”
Asking these standard questions helps us decide where to place the feature, how much tech debt we are willing to incur and how we can build it in a way that doesn’t slow down our UX.
What tools or technologies does your team use to support scalability, and why?
Dealing with different inputs and outputs in various formats can be painful. Luckily, we rely on Kafka to push third-party communications to the fringe of our work. We take an inbound event, transform it and deliver it to the rest of our code in our style without having to scatter processing code for similar yet differently formatted objects.
Good monitoring is another key way to ensure that customers aren’t waiting around. Datadog is an excellent tool that provides great insight into the health and utilization of an environment as well as the logs. It also pairs nicely with AWS as a hosting provider. AWS offers scalable resources and control over costs, making it easier to handle traffic bursts and adjust resources without physical hardware. Choosing a hosting provider is personal, but cloud solutions provide much-needed flexibility.
Valon is a software platform that leverages technology to improve mortgage servicing and homeownership.
Describe what scalability means to you. Why is scalability important for the technology you're building?
Valon’s mortgage servicing product models complex regulatory requirements while moving large sums of money in a reconcilable and auditable manner. As a financial services company, Valon defines scalability as the system’s ability to remain perfectly correct under all conditions as it grows in complexity and volume. Correctness is paramount because Valon faces an incredible amount of scrutiny from all parties that interact with our platform. This includes homeowners who entrust us with managing funds for a crucial financial investment; regulators such as CFPB and FHA who audit us to ensure proper risk management and fair treatment of homeowners; and our clients who reconcile the large sums of money we transact with them on a daily basis.
How do you build this tech with scalability in mind?
Our principles are to minimize system complexity and ensure robustness to meet our business requirements. Valon’s workflows are distributed in nature, which creates a problem regarding distributed transactions. We address this by architecting our system using an eventually consistent model. Our workflows are typically implemented using persisted state machines that are driven to completion via background jobs. The jobs run every N minutes, querying records that have not yet reached a terminal state and transitioning them until they reach the terminal state. Each transition must be fully idempotent — otherwise, there’s a risk of doing duplicate work when retrying a failed transition.
Our principles are to minimize system complexity and ensure robustness to meet our business requirements.”
Separately, we care deeply about well-designed service abstractions and dependencies. We have a service dependency diagram that is enforced by a linter and our platform team is actively driving an effort to draw the right boundaries between service interfaces to reduce complexity.
What tools or technologies does your team use to support scalability, and why?
Our architecture is enabled using a few core components such as MySQL, a custom batch processing framework built on top of Google Pub/Sub, Google Cloud Run for rapidly scaling our workloads up or down, and Grimp, NetworkX and Graphviz for mapping out our service dependency graph.
Enigma provides data on the financial health and identity of small and medium-sized businesses.
Describe what scalability means to you. Why is scalability important for the technology you’re building?
As engineers, we often think of scalability in terms of the processing capabilities of systems under increased load. This is incredibly important. We want to build systems that gracefully incorporate additional resources, have minimal bottlenecks and keep key algorithms near linear.
Equally important is ensuring that teams continue to scale as systems become more complex and handle greater volume. However, as engineers, we give this problem less attention. Our most precious resource is the focus of our engineering talent. Teams with focus can build and extend scalable systems. Teams distracted by keeping their previous builds running often fail to do this.
Teams with focus can build and extend scalable systems. Teams distracted by keeping their previous builds running often fail to do this.”
How do you build this tech with scalability in mind?
Our experience is that seasoned professionals know the best practices for building scalable systems. In less-established companies that are experiencing rapid scaling, the pressure to cut corners can be pervasive. Teams experience the greatest pressure at precisely the points in the build cycle with the most impact on a system’s scalability. Those points are the beginning and the end. Leaders want teams to start the build quickly and then later declare the build complete.
Our job as technical leaders is to resist this pressure. Two indispensable practices include having a transparent, collaborative design process and keeping a balance between non-functional and functional elements of the system.
Somewhat counterintuitively, realistic planning and company culture are essential to building scalable systems. We are most at risk of building systems that don’t scale well when we plan work poorly, fall behind and then respond to pressure by sacrificing elements that would allow the team to scale on future projects.
What tools or technologies does your team use to support scalability, and why?
Choosing the right technologies is necessary for building scalable systems, and a system will only scale as well as the underlying tech it’s built on. That said, to scale well, teams need to be equally thoughtful about how they apply technologies. The tools we use to scale our systems include:
- AWS - Multiple services
- Gitlab - Continuous integration and deployment
- Pagerduty - Production support
- New Relic - Production monitoring of our systems
- Databricks - Compute for our data pipelines
- LakeFS - Data branching
- Dagster - Pipeline orchestration
Hiro Systems creates developer tools for Stacks, the open-source network for Bitcoin contracts and apps.
Describe what scalability means to you. Why is scalability important for the technology you’re building?
In my role, scalability describes how well a service or larger system responds to a change in demand. A simple example would be an API server responding to increased visitor traffic during a big sports game. Inversely, this also means the opposite must also be considered when planning for scalability. A service or system should be able to respond to decreased demand by scaling inward or downward.
A service or system should be able to respond to decreased demand by scaling inward or downward.”
Typically, the primary purpose of scaling outward or upward is to maintain an expected level of throughput, while the purpose of scaling inward or downward is to reduce wasted resources or costs. Like most systems that others depend on, the ones I maintain have service-level agreements and measurable expectations for uptime, performance and cost. Scalability is important to me because it helps keep these three metrics within acceptable levels when demand can change in an instant.
How do you build this tech with scalability in mind?
Before I build or deploy a scalable service, I need to ensure the contextual architecture is able to support such a capability. First and foremost, you need infrastructure that responds to changes in demand and a way to observe and measure that demand. Having one without the other is not going to be nearly as useful.
Services being designed with scalability in mind should be able to export queryable metrics in which the infrastructure can read and make decisions based on the data.
In addition to this, a scalable service should be designed to make full use of the resources given to it and continue to operate correctly when there are multiple replicas of itself, or when one of those replicas is removed. We also consider implementing support for external caches into a service that can provide a great boost to its throughput.
What tools or technologies does your team use to support scalability, and why?
My team packages all of our services in Docker containers and then runs them in Kubernetes, which in turn helps ensure our workloads are horizontally scaled based on their resource and application usage. In conjunction with this, we use Prometheus and Grafana Mimir to collect and store metrics information from our deployed services. Kubernetes, or other services, can then read from those metrics to understand what service is underperforming or low on resources and ultimately decides whether our workloads should be scaled outwards or inwards.
Another tool we use is Knative which enables us to automatically scale a service from zero, keeping our costs as low as possible for certain services until they’re actually needed.
Any services that use databases must be able to distinguish between a read-only database and a read/write database. This allows our data layer to scale as we wish to add or remove database replicas as needed.
To evenly distribute load across running servers, we make use of load balancers at multiple levels once traffic enters our private network. Additionally, we also use Cloudflare’s caching solution and some in-house rate-limiting.
