208 stories
·
8 followers

The Three Stages of Software Engineering

1 Share

As the old engineering adage goes, “Fast, cheap or good: you can only have two.” While individual software developers often face decisions about when and where to make compromises in their code, tech companies as a whole face similar questions on a broader level as well. Things like complexity, speed, organizational alignment, and the availability of resources create restrictions as a company looks to scale software engineering. However, changes to the way you develop, build, test, and release apps can often help mitigate these problems.

Stage One: Simplicity

Scaling software development has to start somewhere. When I joined LinkedIn in 2011, the LinkedIn mobile development team was small—six people small, to be exact. The company only had an iOS app, and it was built on the Three20 framework. We used a Ruby on Rails frontend and our Rails instances interfaced with our backend API. Builds were created on my personal machine and releases were on an as-needed basis. Testing was almost entirely manual and we only tested a few server-side metrics. We had no mobile web presence or Android presence.As you can imagine, this was not the most efficient way to build and deploy software.

One of the most important first steps when starting any major new software development project is to find ways to make the develop-build-release process simple. An early focus on simplicity and straightforward process is key because it allows you to concentrate on creating software that solves your business problem, performs well for your users, and can scale as your user base grows.

Kiran Prasad
Kiran Prasad is a Vice President of Engineering at LinkedIn. He currently leads the Flagship team, which provides LinkedIn's more than 467 million worldwide members with a unified mobile and desktop experience. Prior to joining LinkedIn in 2011, he was the Senior Director of Applications at Palm Inc., where he started and delivered webOS, Palm’s first Windows Phone and was the tech lead for the original Treo.

In the mobile world, many people assume that they have no time to focus on building solid code. They are consequently lured into using tools like PhoneGap, which trade simplicity for portability. But my recommendation is not to do that, because these tools essentially trade technical debt for speed of development and deployment, making it harder for you to understand your own code as your applications evolve. They also make it impossible to use the developer tools, both server side and client side, provided by big platform companies like Apple and Google. These tools can meet the needs of the average project team when you have less than ten or so developers working on a project when it comes to development and testing.

Instead, at the beginning of your scaling basis, you should focus on having specialists that understand your code from the ground up, and avoid trading speed for tools that will complicate the design-build-test process or introduce technical debt in the early days of your project.

Another key thing to avoid during early stages is trying solving problems that don’t move your business forward. Lots of people hear about awesome tool efforts at companies like Google, Facebook, or LinkedIn and want to emulate those companies by building their own custom tooling. But the only reason we and peer organizations build those tools is because we have hundreds or thousands of developers. The number of users and developers that make us have to do that custom tooling, not the other way around. Test Flight, Gradle, the IntelliJ IDE, and other simple tools will meet the majority of your needs while your development team is still small. If someone proposes some custom crazy tooling to address a common engineering problem, then it’s probably the wrong answer.

The performance tradeoffs with our technology stack were causing headaches on a daily basis. We needed to architect our server to make it more I/O friendly, and we needed to fix the testing and release pipeline. By Q3, we had already begun scaling our organization to expand our mobile user base and expand our platform availability. We were at 11 percent of traffic, launching on Android and HTML5-based mobile web and had increased our headcount to 10 engineers. We also switched from a process-based system (Rails) to an evented system (Node.js) for increased performance in handling requests from clients.

Stage Two: Efficiency

Once your engineering organization has grown upwards of 30 people, the next important thing to establish is a sense of release cadence. After we had established a reasonably-sized mobile engineering team at LinkedIn, we ran into a problem with our branch development model, which I call “branch hell.” Essentially, if your releases are far enough apart, developers will try to commit code that isn’t ready for production for fear of missing out on the latest build. This leads to bad code being added to branches, compromises in branch readiness, and ultimately delays in shipping new versions of software.

To address this issue, we ended up adopting a very simple model for releases called the “train model.” In this model, every release goes out to production at a fixed time, just like a train leaving the station. A certain set of features can be on the train or not, but it’s definitely leaving, without question, which solves the issue of delayed releases. When you’re in an early-stage engineering organization, the train can leave whenever you want it to. But once you reach a certain size, you need to create and stick to a schedule—structure around how and when you release new versions.

Focusing on maintaining consistency between platforms will ease development at the tooling level and the cognitive level of understanding the product for everyone.

In order for the train model to work, you need to invest in a trunk-based development model that offers a significantly simplified approach, compared to branch development. Having a good experimentation platform and automated testing suite is a good investment, as well, in order to accelerate your release schedule and increase your test coverage. At this point, I also recommend running against simulators (software emulators and the like) and not focusing on device variability at this stage, until you get more users.

The LinkedIn testing strategy.

The LinkedIn testing strategy.

At LinkedIn, we use a system called LIX, which manages the lifecycle of all of our tests and experiments. Every product feature is behind a LIX test, and functional testing is performed with LIX turned on and off. All check-ins are made directly to the trunk and have to meet our testing criteria in order to ship. When we adopted this system in 2012, it meant that for the first time our automating testing could give us a comprehensive view of our code readiness, which features were ready to ship, and which were not. In short, we had banished “branch hell.”

At this point in our company’s history, we were tracking three primary metrics in order to measure success:

  • The time from design created to code checked in.
  • The time from code checked into code in production.
  • The time from code entering production to it being 100 percent ramped to members.

Once you’re in a more rhythmic execution model, I recommend locking the development of features cross-platform, so that features launched on iOS can be launched on Android as well. This is not just to meet end-user expectations. Rather, diverging the two creates more complexity for everyone, from the designers and project managers to developers. Focusing on maintaining consistency between platforms will ease development at the tooling level and the cognitive level of understanding the product for everyone.

Stage Three: Organizational Alignment

The final stage of scaling any software development project is organizational alignment when the project becomes so big that all of your engineering efforts (infrastructure, databases, operations, etc.) are supporting the same output. This typically happens when you start to get 100’s of developers committing to a single codebase and many millions of users using your software on a daily basis.

As you grow bigger, you actually need to compress the release cycle.

At this scale, you need quality assurance at every stage: pre-commit, post-commit, in production, etc. In many ways, this means you have to give up on the idea that all of your bugs will be caught in your testing cycle, and focus on monitoring the actual behavior of your software in the wild. There are lots of key metrics you can use as signals for how well your product is operating: CPU utilization, DNS lookups, memory utilization, crash rate, etc.: these are all key indicators. It may seem counterintuitive, but there’s no way to monitor what 100 million users are all seeing on their screens, so you actually have to release your software in order to truly understand how it is performing.

Another counterintuitive idea is that as you grow bigger, you actually need to compress the release cycle. At LinkedIn, we went from a once per month release cycle to a 3×3 release cycle. This shorter duration reduces the mega-apprehension that developers might feel the need to chuck a lot of features into a release at the last moment, in order to avoid missing a deadline. With a smaller group of people, you can control this tendency through culture. But when you have a lot of people in many, many different organizations as part of your engineering team, there’s no way you can control that impulse through culture alone. Frequent releases also help developers feel less anxious about missing release deadlines, and as a result, they don’t rush their code into builds.

Having such this kind of release cadence almost necessitates having a fully automated release cycle, simply to keep up with the pace at which your teams are moving. At this stage is when the custom tooling that I discussed earlier comes into play. This is because at this scale, the default tools for iOS, Android and even Web development will be unable to deal with your growing codebase, and will end up slowing down the rate at which your teams can move. They’re simply not built to facilitate that many people are working on the same codebase and features like quality library management are simply not provided by the big platform vendors by default. At LinkedIn, we split our projects between multiple repos to allow for more manageable multi-team collaboration. This necessitated a big investment in custom tooling to make multi-product development work while keeping up with our desired release cadence. Be prepared to make similar investments in your dev infrastructure as you grow your team.

Finally, this section is called “organizational alignment” for a reason. Today at LinkedIn we see mobile as a majority of our business and no longer make the distinction between engineering for mobile vs. desktop. Just as form follows function, your business goals and software projects will also eventually align as the number of users and customers grows. Those users will expect the same experience, regardless of the platform they use. Having two teams that own two different experiences. Fractured experience creates infighting between groups.

Also, you’re not going to keep hiring people into the mobile group if they outnumber the rest of the engineering team, combined. When you project merges with the overall organization as a whole, you can say that you that you truly have “scale” in your engineering organization.

Putting It All Together

Scaling a software engineering organization presents many challenges to the best organizations. It has the potential to introduce costs and complexity, but also presents useful opportunities for improved collaboration and greater operational efficiency.

Often, these challenges are as much about culture and philosophy as they are about process and resources. By reducing organizational fear, increasing the speed of innovation, and creating alignment throughout your organization, you can reap the benefits of a large group of highly-motivated software engineers.

Feature image via Pixabay.

The post The Three Stages of Software Engineering appeared first on The New Stack.

Read the whole story
sashae
2869 days ago
reply
new york, ny
Share this story
Delete

Happy Together

1 Comment

Cole Kennedy moved to New York a year ago. He was twenty-three and had recently graduated from the University of Missouri with a degree in English. He applied for copywriting jobs all over, and assumed that he’d end up in the Midwest. “Moving to New York seemed cool, but it was, like, a thing that happens to other people,” he told me. Then his lottery ticket arrived: a paid internship at Foursquare, the search-and-discovery app, which is based in Manhattan.

See the rest of the story at newyorker.com

Related:
The America’s Cup in Manhattan, Once Again
Car2Go and the Egotism of the Sharing Economy
“Dead Ringer”: A New York City Pay Phone’s Spirited Farewell
Read the whole story
sashae
3117 days ago
reply
remarkable
new york, ny
Share this story
Delete

When Your Lived Reality becomes an Algorithm of the Popular

1 Comment

Yesterday, as people panicked about Twitter’s plan to present tweets to users using algorithmic calculations of what that person might like, I wrote this:

What if humans started to experience time as an algorithm of the popular rather than lived narratively ordered experience?

Especially since I work from home and in flyover country, Twitter is very much a lived conversation for me. And if Twitter alters the way it appears to me — basically choosing who it thinks I want to talk to rather than what the serendipity of the unique collection of people I follow presents in time-ordered fashion — it will be fairly dramatically altering my lived reality.

The thought got even creepier for me as respondents to my tweet (author William Gibson was among those who retweeted it, so I got so really awesome responses) pointed out that associations divorced from lived time is much closer to dreaming than waking reality. Of course, as a shrink friend noted, in dreaming, we consciously and unconsciously select what those associations are, rather than having a computer do it for us.

So thought of as its almost most dystopian, Twitter wants to take the serendipitous global conversation we’ve been having and instead replace it with a living dream world chosen for us algorithmically.

But let me go one step more dystopian. As I noted yesterday, Google recently told the British Parliament that it is testing ways to show “positive” ad words and YouTubes when people look for hateful, potentially terrorist speech. Google’s announcement follows an earlier one from Facebook, stating it would do the same.

In other words, since the early January meeting in Silicon Valley, two of the big tech companies announced plans to rejigger their algorithms selectively for users the algos identify as expressing an interest in terrorism. For those interested in terrorism, Google and Facebook will create a waking dreamworld.

Thus far, Twitter has made no such announcement. Yesterday (that is, the same day this algorithm report came out) it did, however, announce how many perceived terrorists it has kicked off Twitter.

Like most people around the world, we are horrified by the atrocities perpetrated by extremist groups. We condemn the use of Twitter to promote terrorism and the Twitter Rules make it clear that this type of behavior, or any violent threat, is not permitted on our service. As the nature of the terrorist threat has changed, so has our ongoing work in this area. Since the middle of 2015 alone, we’ve suspended over 125,000 accounts for threatening or promoting terrorist acts, primarily related to ISIS.

The blog post making that announcement also addressed algorithms, admitting that they can’t really work, linking to this reportfrom December (which discussed Facebook and Google) on another presentation to the UK Parliament.

As many experts and other companies have noted, there is no “magic algorithm” for identifying terrorist content on the internet, so global online platforms are forced to make challenging judgement calls based on very limited information and guidance. In spite of these challenges, we will continue to aggressively enforce our Rules in this area, and engage with authorities and other relevant organizations to find solutions to this critical issue and promote powerful counter-speech narratives.

All of which might have left the impression that Twitter, unlike its counterparts Google and Facebook, would not be fiddling with its algorithms in response to the request to magnify voices deemed to be positive targeted at those seeking terrorism content.

Except that at almost exactly the same time, came these reports that everyone would get (or would get the option of) the algorithmic treatment.

Now, having studied how, after 1848, the powers that be in Paris found ways to eliminate the growing newspaper public in the belief that it had led to that year’s revolution (that’s partly where the idea of high literature, as embodied in Madame Bovary, arose from), I’m pretty paranoid when I see the ways the elite would current neuter the voices that contributed to the Arab Spring, Black Lives Matter, and yes, terrorism.

But I have to say, I always get buggy when communications companies — like AT&T and Microsoft — don’t have an apparently robust business model, especially when I consider ways they could (and do, in the case of AT&T) profit handsomely off working for the government. Most people assume Twitter is doing this as a way to monetize, which it has thus far failed to do. And that may well be the case.

If so, who is the customer, and what is Twitter delivering?

Update: Jack Dorsey just had this to say on Twitter.

Hello Twitter! Regarding #RIPTwitter: I want you all to know we’re always listening. We never planned to reorder timelines next week.

Twitter is live. Twitter is real-time. Twitter is about who & what you follow. And Twitter is here to stay! By becoming more Twitter-y.

Read the whole story
sashae
3209 days ago
reply
all social networking seems to lead to dystopian control given enough time and scale
new york, ny
Share this story
Delete

This guy. /

3 Comments


This guy. /

Read the whole story
sashae
3267 days ago
reply
classic
new york, ny
Share this story
Delete
2 public comments
Technicalleigh
3267 days ago
reply
I vote to expel this asshole from BusinessTown forever.
Vancouver BC
toddgrotenhuis
3267 days ago
reply
too soon?
Indianapolis
srsly
3267 days ago
never

Totes had to post this new character #blessed #gratitude...

2 Comments and 3 Shares


Totes had to post this new character #blessed #gratitude #hashtags /

Read the whole story
sashae
3284 days ago
reply
ugh lord so true
new york, ny
Share this story
Delete
1 public comment
Technicalleigh
3285 days ago
reply
BUSINESSTOWN FOREVER.
Vancouver BC

The We-Branding of Downtown Manhattan

1 Comment

corbin

corbin

The new Fulton Street Transit Center, where nine train lines come together beneath the glittering oculus of the so-called “Sky-Reflector Net,” opened a little over a year ago to much fanfare more than a decade after part of it was destroyed in the terrorist attacks of September 11, 2001. “New Yorkers, accustomed to thinking of transit hubs like Penn Station and Times Square as places to suffer through, will find on Monday morning a kind of Crystal Palace, crowned by a dome that funnels daylight two stories below ground,” the New York Times wrote of its reopening. Adjacent to the Fulton Center, and providing a street-level entrance to the station is the nine-story Corbin Building.

This summer, the Landmarks Preservation Commission designated the Corbin Building, constructed from 1888 to 1889, a landmark. But when the MTA announced its plans to build a new transit center, it was in sorry enough shape to come under threat of demolition until, in October 2003, the MTA committed to including the building in its vision. The Corbin Building is named for developer and financier Austin Corbin, who died shortly after a violent carriage accident at his house in New Hampshire and is best known now for his management and stewardship of the Long Island Railroad. It was designed by Frances Hatch Kimball, who, in a 1917 story noting his declaration of bankruptcy, the Times described, possibly with some irony, as the “‘Father of the Skyscraper’ and Designer of Notable Buildings.” Ultimately, the Corbin Building’s restoration accounted for over $67 million of the $1.5 billion transit center budget.

Having been restored, the Corbin Building now offered some thirty-eight thousand square feet of empty commercial space in downtown Manhattan, right next door to one of the city’s biggest transit hubs, and just down the street from the new World Trade Center. But the MTA is not (necessarily) in the business of building management, and in 2012 it issued a request for proposals, inviting private developers to submit plans for how they would put that space to use, as well as another thirty-thousand square feet of commercial space in the transit center itself.

In December 2013, the MTA announced that an Australian company called the Westfield Group had been chosen as the master lessee. (Last year, the Westfield Group restructured into two, independent companies—the Westfield Corporation now runs things outside of Australia and New Zealand.) About a year later, in the first quarter of financial year 2015 (just around the time that the new Fulton Street Transit Center opened), Westfield leased the Corbin Building to WeWork, a co-working startup based in New York. WeWork signed a fifteen-year lease for thirty-eight thousand square feet across all nine floors. WeWork does not have to pay rent for the first eleven months. This isn’t unusual for transaction of this size, but it does fit into WeWork’s larger business strategy—recently illuminated by both The Information and BuzzFeed—of pushing off expenses as far into the future as possible, while betting, with its $10 billion valuation, that it can attract sufficient membership to eventually meet those (fixed) costs. After the first eleven months, its rent is $50 per square foot for the first five years, $55 for the next five years, and $60 for the next. The net effective rent, then, over the course of the fifteen-year lease, is $51.74 per square foot—a few dollars higher than the average effective rent in the area. (A pretty good deal for the Westfield Corporation, then!) WeWork has heavily invested in the area around Fulton Center: It already has office space at 222 Broadway, and, in 2013, WeWork co-founder Adam Neumann acquired the rights to buy twenty-five floors in the Woolworth Building, at 233 Broadway, for $68 million.

In its development proposal, submitted in response to the MTA’s 2012 RFP, and which I obtained by a FOIL request, the Westfield Group said that it would lease the first three levels of the Corbin Building to retailers and the remaining six as office space, at a rate of two floors per year, starting in 2014. “Therefore, based on current Lower Manhattan office market conditions, we have identified 2017 as our stabilized year (fully leased),” the proposal reads. It would seem things are ahead of schedule.

The first three floors were primarily to be used for luxury women’s accessories, Westfield proposed: “Under Westfield’s plan, this space will be utilized for one or up to three signature specialty retailers who will hold a very special Broadway address and truly unique space.” The office space, meanwhile, was envisioned as “a home to a vibrant mix of ambitious technology firms, media companies, small financial institutions and boutique architectural and law firms,” according to the development proposal. “Such organizations seek a distinct address and Corbin’s historic jewel-box building clearly presents that opportunity. This is where maturing companies will converge to take advantage of a rare state-of-the-art office building with twenty-four seven (24/7) operational capabilities and direct access to major public transportation lines. The unique loft-like setting with its high ceilings and natural light, coupled with connections to great dining options within the Fulton Center, will create one of the most desirable small-user office spaces in Lower Manhattan.”

Leasing the office space to WeWork is not exactly a departure from this plan, as it is conceivable that WeWork, which largely specializes in subletting over-priced and over-amenitized office space to over-valued startups, could bring in a “vibrant mix” of tenants. What’s less clear is whether WeWork is going to get into retail, or whether that part of the plan has been abandoned. In May, the Tribeca Citizen reported that WeWork applied to Community Board 1 for several liquor licenses under several different LLCs: at the Corbin Building, as WeWork Fulton Center LLC; at 110 Wall Street—where its secretive WeLive project is being explored—as WW Service LLC and The Mailroom LLC; and at 85 Broad Street, as WeWork Social Club LLC. The party never stops. Through a spokesperson, WeWork declined to comment, as it always does. The MTA, through a spokesperson, referred questions about the lease to Westfield, which, through a spokesperson, also declined to comment.

Westfield is a ubiquitous presence in the post-9/11, downtown cityscape. In addition to being awarded control of the retail and commercial space of the sprawling Fulton Center, Westfield also paid the Port Authority of New York and New Jersey $800 million for full control of retail development at the $3.7 billion World Trade Center Transportation Hub, which has been repeatedly delayed by water leaks, and which will also house the $1.4 billion Westfield World Trade Center shopping center, “the most complete retail destination in New York City, the most alluring retail landmark in the world.”

But what Westfield does isn’t just about bringing in retailers and restaurants: It is about creating Dynamic Brand Experiences From Morning Till Night: “Conceived as a high-traffic hub of connectivity, Fulton Center moves commuters, residents, and tourists seamlessly throughout major subway lines and the city—creating targeted opportunities for brand engagements day and night.” What differentiates Westfield from other developers, according to its proposal to the MTA, is “its ability to connect consumers with the best brands, creating communications that lead to a relationship.” Westfield is committed to “delivering one-of-a-kind value propositions by creating ‘Complete Brand Experiences’ and ‘Domination Packages’ allowing brands to govern an entire visual landscape with style and sophistication.”

As always, the MTA’s decision to grant Westfield the master lease at Fulton Center, and the Port Authority’s decision to grant them control of retail at the World Trade Center, is ultimately about money. “We expect to generate enough revenue to maintain the whole complex,” the president of MTA’s Capital Construction division, Michael Horodniceanu, told the Times after the RFP went out. “It would be interesting if we could get a bank as the anchor tenant.” (Austin Corbin’s Corbin Banking Company was the ground floor’s original occupant.) Westfield’s play is to leverage the three hundred thousand people who pass through Fulton Center each day and the five hundred thousand people who will pass through the forthcoming World Trade Center transit hub as not just commuters but potential customers, waiting, at each turn taken through the subterranean labyrinth on their way to their jobs—some just up the steps and around the corner, the creators and makers and entrepreneurs filling the Corbin Building—to be sold something.

Photo by MTA

Read the whole story
sashae
3288 days ago
reply
"creating targeted opportunities for brand engagement day and night" ugh
new york, ny
Share this story
Delete
Next Page of Stories