As the old engineering adage goes, “Fast, cheap or good: you can only have two.” While individual software developers often face decisions about when and where to make compromises in their code, tech companies as a whole face similar questions on a broader level as well. Things like complexity, speed, organizational alignment, and the availability of resources create restrictions as a company looks to scale software engineering. However, changes to the way you develop, build, test, and release apps can often help mitigate these problems.

Stage One: Simplicity

Scaling software development has to start somewhere. When I joined LinkedIn in 2011, the LinkedIn mobile development team was small—six people small, to be exact. The company only had an iOS app, and it was built on the Three20 framework. We used a Ruby on Rails frontend and our Rails instances interfaced with our backend API. Builds were created on my personal machine and releases were on an as-needed basis. Testing was almost entirely manual and we only tested a few server-side metrics. We had no mobile web presence or Android presence.As you can imagine, this was not the most efficient way to build and deploy software.

One of the most important first steps when starting any major new software development project is to find ways to make the develop-build-release process simple. An early focus on simplicity and straightforward process is key because it allows you to concentrate on creating software that solves your business problem, performs well for your users, and can scale as your user base grows.

Kiran Prasad

Kiran Prasad is a Vice President of Engineering at LinkedIn. He currently leads the Flagship team, which provides LinkedIn's more than 467 million worldwide members with a unified mobile and desktop experience. Prior to joining LinkedIn in 2011, he was the Senior Director of Applications at Palm Inc., where he started and delivered webOS, Palm’s first Windows Phone and was the tech lead for the original Treo.

In the mobile world, many people assume that they have no time to focus on building solid code. They are consequently lured into using tools like PhoneGap, which trade simplicity for portability. But my recommendation is not to do that, because these tools essentially trade technical debt for speed of development and deployment, making it harder for you to understand your own code as your applications evolve. They also make it impossible to use the developer tools, both server side and client side, provided by big platform companies like Apple and Google. These tools can meet the needs of the average project team when you have less than ten or so developers working on a project when it comes to development and testing.

Instead, at the beginning of your scaling basis, you should focus on having specialists that understand your code from the ground up, and avoid trading speed for tools that will complicate the design-build-test process or introduce technical debt in the early days of your project.

Another key thing to avoid during early stages is trying solving problems that don’t move your business forward. Lots of people hear about awesome tool efforts at companies like Google, Facebook, or LinkedIn and want to emulate those companies by building their own custom tooling. But the only reason we and peer organizations build those tools is because we have hundreds or thousands of developers. The number of users and developers that make us have to do that custom tooling, not the other way around. Test Flight, Gradle, the IntelliJ IDE, and other simple tools will meet the majority of your needs while your development team is still small. If someone proposes some custom crazy tooling to address a common engineering problem, then it’s probably the wrong answer.

The performance tradeoffs with our technology stack were causing headaches on a daily basis. We needed to architect our server to make it more I/O friendly, and we needed to fix the testing and release pipeline. By Q3, we had already begun scaling our organization to expand our mobile user base and expand our platform availability. We were at 11 percent of traffic, launching on Android and HTML5-based mobile web and had increased our headcount to 10 engineers. We also switched from a process-based system (Rails) to an evented system (Node.js) for increased performance in handling requests from clients.

Stage Two: Efficiency

Once your engineering organization has grown upwards of 30 people, the next important thing to establish is a sense of release cadence. After we had established a reasonably-sized mobile engineering team at LinkedIn, we ran into a problem with our branch development model, which I call “branch hell.” Essentially, if your releases are far enough apart, developers will try to commit code that isn’t ready for production for fear of missing out on the latest build. This leads to bad code being added to branches, compromises in branch readiness, and ultimately delays in shipping new versions of software.

To address this issue, we ended up adopting a very simple model for releases called the “train model.” In this model, every release goes out to production at a fixed time, just like a train leaving the station. A certain set of features can be on the train or not, but it’s definitely leaving, without question, which solves the issue of delayed releases. When you’re in an early-stage engineering organization, the train can leave whenever you want it to. But once you reach a certain size, you need to create and stick to a schedule—structure around how and when you release new versions.

Focusing on maintaining consistency between platforms will ease development at the tooling level and the cognitive level of understanding the product for everyone.

In order for the train model to work, you need to invest in a trunk-based development model that offers a significantly simplified approach, compared to branch development. Having a good experimentation platform and automated testing suite is a good investment, as well, in order to accelerate your release schedule and increase your test coverage. At this point, I also recommend running against simulators (software emulators and the like) and not focusing on device variability at this stage, until you get more users.

The LinkedIn testing strategy.

At LinkedIn, we use a system called LIX, which manages the lifecycle of all of our tests and experiments. Every product feature is behind a LIX test, and functional testing is performed with LIX turned on and off. All check-ins are made directly to the trunk and have to meet our testing criteria in order to ship. When we adopted this system in 2012, it meant that for the first time our automating testing could give us a comprehensive view of our code readiness, which features were ready to ship, and which were not. In short, we had banished “branch hell.”

At this point in our company’s history, we were tracking three primary metrics in order to measure success:

• The time from design created to code checked in.
• The time from code checked into code in production.
• The time from code entering production to it being 100 percent ramped to members.

Once you’re in a more rhythmic execution model, I recommend locking the development of features cross-platform, so that features launched on iOS can be launched on Android as well. This is not just to meet end-user expectations. Rather, diverging the two creates more complexity for everyone, from the designers and project managers to developers. Focusing on maintaining consistency between platforms will ease development at the tooling level and the cognitive level of understanding the product for everyone.

Stage Three: Organizational Alignment

The final stage of scaling any software development project is organizational alignment when the project becomes so big that all of your engineering efforts (infrastructure, databases, operations, etc.) are supporting the same output. This typically happens when you start to get 100’s of developers committing to a single codebase and many millions of users using your software on a daily basis.

As you grow bigger, you actually need to compress the release cycle.

At this scale, you need quality assurance at every stage: pre-commit, post-commit, in production, etc. In many ways, this means you have to give up on the idea that all of your bugs will be caught in your testing cycle, and focus on monitoring the actual behavior of your software in the wild. There are lots of key metrics you can use as signals for how well your product is operating: CPU utilization, DNS lookups, memory utilization, crash rate, etc.: these are all key indicators. It may seem counterintuitive, but there’s no way to monitor what 100 million users are all seeing on their screens, so you actually have to release your software in order to truly understand how it is performing.

Another counterintuitive idea is that as you grow bigger, you actually need to compress the release cycle. At LinkedIn, we went from a once per month release cycle to a 3×3 release cycle. This shorter duration reduces the mega-apprehension that developers might feel the need to chuck a lot of features into a release at the last moment, in order to avoid missing a deadline. With a smaller group of people, you can control this tendency through culture. But when you have a lot of people in many, many different organizations as part of your engineering team, there’s no way you can control that impulse through culture alone. Frequent releases also help developers feel less anxious about missing release deadlines, and as a result, they don’t rush their code into builds.

Having such this kind of release cadence almost necessitates having a fully automated release cycle, simply to keep up with the pace at which your teams are moving. At this stage is when the custom tooling that I discussed earlier comes into play. This is because at this scale, the default tools for iOS, Android and even Web development will be unable to deal with your growing codebase, and will end up slowing down the rate at which your teams can move. They’re simply not built to facilitate that many people are working on the same codebase and features like quality library management are simply not provided by the big platform vendors by default. At LinkedIn, we split our projects between multiple repos to allow for more manageable multi-team collaboration. This necessitated a big investment in custom tooling to make multi-product development work while keeping up with our desired release cadence. Be prepared to make similar investments in your dev infrastructure as you grow your team.

Finally, this section is called “organizational alignment” for a reason. Today at LinkedIn we see mobile as a majority of our business and no longer make the distinction between engineering for mobile vs. desktop. Just as form follows function, your business goals and software projects will also eventually align as the number of users and customers grows. Those users will expect the same experience, regardless of the platform they use. Having two teams that own two different experiences. Fractured experience creates infighting between groups.

Also, you’re not going to keep hiring people into the mobile group if they outnumber the rest of the engineering team, combined. When you project merges with the overall organization as a whole, you can say that you that you truly have “scale” in your engineering organization.

Putting It All Together

Scaling a software engineering organization presents many challenges to the best organizations. It has the potential to introduce costs and complexity, but also presents useful opportunities for improved collaboration and greater operational efficiency.

Often, these challenges are as much about culture and philosophy as they are about process and resources. By reducing organizational fear, increasing the speed of innovation, and creating alignment throughout your organization, you can reap the benefits of a large group of highly-motivated software engineers.

Feature image via Pixabay.

The post The Three Stages of Software Engineering appeared first on The New Stack.

Cole Kennedy moved to New York a year ago. He was twenty-three and had recently graduated from the University of Missouri with a degree in English. He applied for copywriting jobs all over, and assumed that he’d end up in the Midwest. “Moving to New York seemed cool, but it was, like, a thing that happens to other people,” he told me. Then his lottery ticket arrived: a paid internship at Foursquare, the search-and-discovery app, which is based in Manhattan.

See the rest of the story at newyorker.com

Yesterday, as people panicked about Twitter’s plan to present tweets to users using algorithmic calculations of what that person might like, I wrote this:

What if humans started to experience time as an algorithm of the popular rather than lived narratively ordered experience?

Especially since I work from home and in flyover country, Twitter is very much a lived conversation for me. And if Twitter alters the way it appears to me — basically choosing who it thinks I want to talk to rather than what the serendipity of the unique collection of people I follow presents in time-ordered fashion — it will be fairly dramatically altering my lived reality.

The thought got even creepier for me as respondents to my tweet (author William Gibson was among those who retweeted it, so I got so really awesome responses) pointed out that associations divorced from lived time is much closer to dreaming than waking reality. Of course, as a shrink friend noted, in dreaming, we consciously and unconsciously select what those associations are, rather than having a computer do it for us.

So thought of as its almost most dystopian, Twitter wants to take the serendipitous global conversation we’ve been having and instead replace it with a living dream world chosen for us algorithmically.

But let me go one step more dystopian. As I noted yesterday, Google recently told the British Parliament that it is testing ways to show “positive” ad words and YouTubes when people look for hateful, potentially terrorist speech. Google’s announcement follows an earlier one from Facebook, stating it would do the same.

In other words, since the early January meeting in Silicon Valley, two of the big tech companies announced plans to rejigger their algorithms selectively for users the algos identify as expressing an interest in terrorism. For those interested in terrorism, Google and Facebook will create a waking dreamworld.

Thus far, Twitter has made no such announcement. Yesterday (that is, the same day this algorithm report came out) it did, however, announce how many perceived terrorists it has kicked off Twitter.

Like most people around the world, we are horrified by the atrocities perpetrated by extremist groups. We condemn the use of Twitter to promote terrorism and the Twitter Rules make it clear that this type of behavior, or any violent threat, is not permitted on our service. As the nature of the terrorist threat has changed, so has our ongoing work in this area. Since the middle of 2015 alone, we’ve suspended over 125,000 accounts for threatening or promoting terrorist acts, primarily related to ISIS.

The blog post making that announcement also addressed algorithms, admitting that they can’t really work, linking to this reportfrom December (which discussed Facebook and Google) on another presentation to the UK Parliament.

As many experts and other companies have noted, there is no “magic algorithm” for identifying terrorist content on the internet, so global online platforms are forced to make challenging judgement calls based on very limited information and guidance. In spite of these challenges, we will continue to aggressively enforce our Rules in this area, and engage with authorities and other relevant organizations to find solutions to this critical issue and promote powerful counter-speech narratives.

All of which might have left the impression that Twitter, unlike its counterparts Google and Facebook, would not be fiddling with its algorithms in response to the request to magnify voices deemed to be positive targeted at those seeking terrorism content.

Except that at almost exactly the same time, came these reports that everyone would get (or would get the option of) the algorithmic treatment.

Now, having studied how, after 1848, the powers that be in Paris found ways to eliminate the growing newspaper public in the belief that it had led to that year’s revolution (that’s partly where the idea of high literature, as embodied in Madame Bovary, arose from), I’m pretty paranoid when I see the ways the elite would current neuter the voices that contributed to the Arab Spring, Black Lives Matter, and yes, terrorism.

But I have to say, I always get buggy when communications companies — like AT&T and Microsoft — don’t have an apparently robust business model, especially when I consider ways they could (and do, in the case of AT&T) profit handsomely off working for the government. Most people assume Twitter is doing this as a way to monetize, which it has thus far failed to do. And that may well be the case.

If so, who is the customer, and what is Twitter delivering?

Update: Jack Dorsey just had this to say on Twitter.

Hello Twitter! Regarding #RIPTwitter: I want you all to know we’re always listening. We never planned to reorder timelines next week.

Twitter is live. Twitter is real-time. Twitter is about who & what you follow. And Twitter is here to stay! By becoming more Twitter-y.

This guy. /

Totes had to post this new character #blessed #gratitude #hashtags /