The strategy had a number of merits — it brought two recently merged engineering cultures closer together and gave us a shared focus during a complex migration period. But it wasn't the decision-making tool I'd hoped for, and I knew we could do better.

When it came time to start our strategy refresh, those lessons shaped every decision about how we approached it.

What we did differently

This time we had a clear company strategy to anchor to. Storio2030 (our company vision) gave us the end state that was missing the first time around. That meant the technology strategy could focus on the "how" rather than defining the "why" or the "what" — those were already established.

We also made a deliberate choice not to build a functional strategy. Technology at Storio is an enabler for almost everything the business does. A strategy created by technology for technology would repeat the same mistake. This needed to be an organisational enablement strategy with input, challenge and feedback woven in from across the business.

Finally we were clear up front about what success looked like. Not a document that sat on a shelf but a strategy that enabled better and faster decisions by the teams closest to the work. That meant the output needed to be accessible, actionable and something people could engage with without a thirty-page read.

The vision I shared internally

To kick off the process I wrote a foreword for the organisation framing why this strategy matters and how I wanted us to approach it. Here it is as I shared it.

I first started writing code to solve business problems in the year 2000, during the dot com boom. I was frustrated with having to manually aggregate electricity meter data from thousands of sources to accurately trade energy. It quickly became clear to me that leveraging technology for efficient solutions would soon become the standard approach for every business.

Fast forward 25 years and that prediction has been proved right. Sure, the technology has moved on quite a lot — from Excel VBA macros to LLM-generated code — but finding efficient and scalable solutions to enable happier customers and successful business outcomes is still the fundamental reason why we exist as a Technology function. Technology is an enabler to almost every facet of Storio. This strategy aims to ensure success through aligned context, a clear vision for where we are going, and repeatable ways of both finding and solving the biggest blockers and opportunities on our path to Storio2030.

This strategy will primarily be focused on the "How" Storio approaches building technology. The "Why" is derived from our company vision, and the "What" is defined by our core business outcomes: the Storio2030 vision and other strategic themes such as our Digital Experience Strategy (a cross-functional strategy defining our customer-facing product direction). Crucially, this is not a siloed Technology functional strategy; it's an organisational enablement strategy. My ambition is for it to influence and empower everyone who uses technology across Storio — Product, Operations, Commercial, Marketing, Finance and HR. There will be functional elements, but this strategy's creation process cannot happen in a vacuum. We need to actively weave input, challenge, and feedback from the entire organisation.

Storio was created during the Web 2.0 era, enabled by the seismic shift from film to digital. At the same time web capability improvements removed friction — no longer did you need to download an application — significantly widening our customer base. AI is poised to enable the next, equally significant capability shift. This strategy will ensure that once again Storio can capitalise on a technology shift to significantly accelerate our ambitions.

Ultimately, this strategy's success won't be measured by its documentation, but by the quality of the decisions it enables to make Storio2030 a reality. Those decisions must be made quickly and efficiently by the teams closest to the work. To achieve this, the strategy needs to be created in a way that enables wide input and challenge to create a clear vision and actionable tenets. This will be both top-down and bottom-up, combining our ambitious aspirations with the practical reality of where Storio is today.

Why this framing mattered

Sharing this foreword was deliberate. It set the tone for how the strategy would be created — not handed down from technology but built collaboratively. It gave people permission to challenge and contribute from day one.

The impact showed up quickly. Working group conversations shifted from "what does technology want to do?" to "what do we need to enable?" and the engagement at our offsite showed people felt genuine ownership of the direction. That was the signal we were on the right track.

In future posts I'll share the structure that emerged from this process — the framework we used to turn this vision into actionable tenets and how we kept the organisation involved throughout.

Before we kicked off a strategy refresh I first spent time reflecting on why. I want to share those lessons here because I think they apply well beyond my own experience and because they have fundamentally shaped how I approach strategy now.

The context

Around six months into the role I kicked off my first strategy process. Storio had recently been through a merger and every function was tasked with creating a vision for their future state. I also had two completely separate engineering organisations, each accountable for their own systems. Bringing them together under a shared direction felt like the right first move.

It was — and the process taught me a lot about what I'd do differently next time.

No company strategy to anchor to

At the time the company's long term strategy was still being developed. That meant I was building a technology strategy in isolation, without a clear end state to work towards.

Strategy should be reductive. It should make decisions easier by giving people a lens to evaluate options against. Without a guiding company view that lens doesn't exist and technology becomes a function optimising for its own goals rather than enabling the business.

The lesson: don't build a functional strategy ahead of the company strategy it needs to serve. If the business direction isn't clear yet, focus on foundations and how to enable future decision making.

Not enough time understanding the current state

We moved too quickly into defining the future without making sure everyone was on the same page about today. And by everyone I don't just mean the technology function. I mean the executive team and the other functions that depend on technology every day.

This gap didn't show itself immediately. It surfaced when we started implementing and realised that the wider organisation wasn't aligned with the need for change or the reasoning behind it.

The lesson: invest in current state alignment before you define the future. Make sure the people who will need to support and fund the change understand and agree on the problems you are solving.

Shorter strategies are easier to use

Without clear outcomes defined up front we ended up producing a twenty-page strategy document. The debates that went into it were genuinely valuable but the artefact itself was too detailed for anyone outside the process to engage with. It required too much time investment to read, let alone act on.

The specificity also worked against us. We had come from a highly autonomous culture with a diverse technology stack. Being prescriptive about solutions meant every recommendation carried a huge cost of change across different teams and contexts.

The lesson: define how the strategy will be used in practice before you start creating it. If the goal is organisational alignment then the format needs to be accessible, concise and flexible enough to accommodate different starting points.

Too insular to drive real change

This was a technology strategy built by the technology function. The problem is that technology at Storio is an enabler for almost everything the business does. Delivering on our strategy required changes across multiple functions, not just our own.

Because it was created in isolation it lacked the cross-functional buy-in needed to drive those wider changes. The size, format and complexity of the document compounded this by making it hard to socialise beyond our own teams.

The lesson: if your function is an enabler then your strategy cannot live inside your function alone. The people who need to change their behaviour to make it real need to be part of creating it.

What this taught me

Looking back, these challenges share a common thread. Each one was a symptom of building in a silo. Siloed from company strategy, from shared understanding of the current state, from how the output would be used, and from the people who needed to act on it.

The strategy wasn't without value — it played a real role in bringing two recently merged engineering cultures together, especially as we navigated complex migrations. But I knew we could go further, particularly in making it a tool that drove decisions and enabled change across the whole organisation.

These learnings have fundamentally shaped how I think about creating strategy and what I believe makes one successful. In my next post I share how these lessons shaped the framing for our strategy refresh — and the internal vision that kicked it off.

Overnight my engineering scope doubled and expanded to include Data, IT and Security for the first time. I became accountable for two different organisations, Albelli the company we merged with was based in Amsterdam which brought cultural differences I hadn't anticipated. For the first six months all I wanted to do was not make any mistakes. Especially with our first peak season coming where we made a significant amount of our profit.

Finding My Voice on the Executive Team

Stepping into the Executive team broadened my focus and I was forced to engage in a number of topics that previously I hadn't. This was one of my first challenges. As a technology leader how much should I engage with commercial, operational and strategic topics that I had limited context on?

There is no right answer to this but my personal value is to only speak when I am adding value. It is easy to feel like you need to have an opinion on everything when you join a senior leadership team. I chose not to and still take this approach to this day. It took time to build the context needed to contribute meaningfully outside of technology and I think that patience paid off.

The Importance of Face to Face

With a distributed workforce one of my first tasks was to get around to all of our locations and meet everyone. This can be quite daunting for someone on the introverted side of the personality spectrum but it was hugely important.

Human connections forged face to face are so beneficial when you are going through change. Not because of the small talk but because you need to find the people who will give you honest feedback. The ones who will tell you what is actually going on rather than what they think you want to hear. Those relationships became invaluable and I could not have built them over video calls alone.

The Culture Lesson That Changed Everything

One of my bigger challenges was navigating culture. Country culture, company culture and functional culture all intertwined with each other and I underestimated the complexity of this.

My biggest insight came from a Principal Engineer about six months in. I had really felt like I was being ignored in a number of areas and could not figure out why. Digging into it I learned that my default approach of giving my opinion first and then asking for others to challenge and adapt based on logic was not working. Culturally it felt like I was telling everyone what to do. So it was ignored.

Simply by inverting that approach to asking for others opinions first and then sharing mine the impact I was able to have changed dramatically. It sounds so obvious written down but when it is your default communication style it is a hard thing to recognise without someone being honest enough to tell you.

Setting the Direction

Things really started to click when we began writing our first Technology Strategy. Rather than debating approaches in meetings, we wrote down our processes, principles and cultural expectations for how we work with technology. Having it on paper meant both teams could engage with the ideas without it feeling like one side was imposing on the other. It depersonalised the conversation and began the long process of standardisation.

That foundation helped us navigate our first peak season as a merged entity successfully. We made minimal changes and focused on validating load and features before the traffic spikes. It was not glamorous but it worked.

The process of creating that strategy was genuinely valuable for bringing people together. The strategy itself? That's a different story.

This isn't a tutorial. There are plenty of WordPress-to-static migration guides out there. This is more of a reflection on what it's actually like to use an AI coding agent for a real, messy, end-to-end project — and what I learned about how to work with it effectively.

The Starting Point

The blog was running on WordPress, hosted on AWS EC2. The content spanned 2012 to 2019 — technical posts about scaling teams, WordPress infrastructure, product development, and engineering leadership. Many posts had embedded images, some had SoundCloud embeds, a couple had image galleries. The URLs had been shared and linked to over the years, so preserving them was non-negotiable.

We chose Eleventy as the target because it's lightweight, flexible, and I didn't need a framework — just a static site generator that could take markdown files and produce HTML.

What Actually Happened

The whole migration took about a week of evening sessions. There were several distinct pieces of work, each with their own challenges. The best part of this was it was like working with an agency doing all of the work in the background. I was able to use my extensive experience to break down the process into chunks and then outsource the doing to Claude. Humans doing the thinking with AI doing the doing is going to become more prevalent in the coming weeks, months and year. Even better the support I got wasn't just for coding but for design, tagging and performance.

Extracting the content. I had a MySQL dump of the WordPress database, so the first job was parsing that to extract the published posts. Claude built a SQL parser in Node.js that could handle WordPress's escaped content, then a migration script that converted the HTML to markdown using Turndown. This gave us the raw material — fifty-five posts converted to markdown, each with structured metadata at the top of the file that tells the site generator how to render it — the title, date, URL, and so on.

Handling the WordPress quirks. This was the bulk of the work. WordPress has years of accumulated implicit behaviour, and migrating content faithfully means dealing with all of it. SoundCloud shortcodes needed converting to iframe embeds. Caption shortcodes wrapping images needed unwrapping. Heading levels were inconsistent across posts and needed normalising. The wpautop function that WordPress uses to add paragraph tags created formatting issues when converted to markdown — walls of text where there should have been paragraphs. Image galleries were particularly fiddly, stored in a WordPress-specific shortcode format that needed converting to a CSS-only lightbox grid.

Migrating images. Hundreds of images were still hosted on the old WordPress CDN. Claude built a download script that pulled each image, saved it locally with the right directory structure, and updated the markdown references to point to the local paths. This ran post-by-post so we could verify nothing was missed.

Preserving URLs. The original WordPress URLs had been shared and linked to over the years. Every single one needed to work on the new site. This meant matching Eleventy's permalink structure to the old WordPress slugs and validating that all fifty-five URLs resolved correctly.

Building the validation suite. A comprehensive script that checked URL preservation, image integrity, internal links, HTML meta tags, RSS feed entries, and sitemap coverage. This became the safety net for the whole project — every change could be verified automatically against it.

Deployment and infrastructure. A GitHub Actions pipeline that builds the site, runs validation, syncs to S3, and invalidates the CloudFront cache. Plus the SEO fundamentals: Open Graph tags, canonical URLs, sitemap, robots.txt, and a 404 page.

By the end: 55 posts migrated, 138 images preserved, all original URLs maintained, automated deployment, and a validation suite that gave me confidence nothing was broken.

What I Learned About Working with AI Agents

Batch and iterate, don't try to do everything at once

The most effective pattern was migrating posts in small batches — typically five at a time — visually checking the output, then fixing whatever was wrong before doing the next batch. Each batch surfaced new edge cases, and fixing them improved the migration script for all subsequent batches.

When I tried to do too many posts at once, the problems compounded and it was harder to see what had gone wrong. Small batches with fast feedback loops worked much better.

Invest in validation early

The validation script was one of the best decisions of the project. Once it existed, every subsequent change could be verified automatically. It checked things I would never have manually tested — every single URL preserved, every image reference resolving, every post appearing in the RSS feed.

Building that kind of safety net early freed me up to move faster on everything else. It's the same principle as writing tests before refactoring, applied to a migration.

The AI is excellent at the tedious parts

Parsing WordPress SQL dumps. Converting shortcodes. Downloading and re-referencing hundreds of images. Generating consistent content structures. These are the kinds of tasks that would have taken hours of manual effort and been incredibly error-prone. Claude handled them reliably and fast.

But you need to stay in the loop on decisions

The places where things went wrong were almost always about context that I hadn't communicated. WordPress has a lot of implicit behaviour — the way wpautop adds paragraph tags, the way galleries are stored, the way internal links are structured. Claude doesn't know your specific WordPress setup unless you tell it, and even then, some things only become apparent when you look at the rendered output.

The most productive pattern was: let Claude do the implementation, but review the output yourself. Not the code — the actual rendered blog posts in the browser.

The long tail is where the time goes

The initial migration — getting posts from WordPress into markdown — was maybe 20% of the total effort. The other 80% was everything else: fixing formatting edge cases, handling embeds, getting images right, setting up deployment, SEO, validation. This is true of most software projects, but it's worth remembering when you're estimating how long an AI-assisted project will take.

Would I Do It Again This Way?

Without question. What would have been a tedious multi-weekend project became a fun background task across a week. The migration scripts, the validation suite, the deployment pipeline — none of these are things I'd have built if I were doing it manually. I'd have done it by hand, made mistakes, and not had the safety net to catch them.

The experience also changed how I think about using AI for engineering work more broadly. The key insight isn't that AI can write code — it's that AI changes what's worth automating. Tasks that would previously have been too small to justify writing a script for suddenly become worth scripting, because the cost of creating that automation has dropped so dramatically.

That's the real shift. Not "AI writes code for me" but "the economics of automation have fundamentally changed." And I think we're still in the very early days of understanding what that means.

Claude also wrote this post for me (promise that this won't be the case in the future) based on the existing content and our conversations together.

It was one the most difficult professional decisions that I have made, due to the depth which I had sunk into the problem space and the amazing people that I was working with. However my desire was to learn how to run multiple high performing teams concurrently at scale and I had confidence that the team at The Tab was strong enough to do it without me.

I started Photobox on 1st August 2016, since then I have been iterating on people, process and principles. With a goal to enable over 70 people embedded in more than 10 teams to have maximum impact within a private equity investment cycle. By no means has this journey been straight forward or is it finished, however I feel like it is a good time reflect on my thoughts so far.

Start with principles

Starting with a set of principles which explain the how and why is a good way to begin a process of alignment around change.

High trust environments enable more effective change

Conversations around change are best done in a safe space with high trust. My experience is that this most often exists already within existing team structures. You get much better feedback and can tailor responses to the teams context which keeps the conversation relevant and engaged.

Start small and iterate

Start small and iterate applies to teams, org structures and operating models as much as it does to releasing software. Big bang changes tend to need the most support and have potential to cause the most negative impact. We have now moved to a model where we change one team at a time and iterate with learning into the next one. This has allowed us to refine the change process as we go and make sure to embed feedback and build out good content for wider sharing.

Capacity for change is not uniform

Change impacts different people and cultures in different ways. When change is constant people can become stuck in the trough of uncertainty and this has a massive impact on their ability to have impact. Understanding that change impacts me in a different way to others helped me take a much more empathetic approach.

Write things down

Writing how you want to work down and then debating it with wider and wider circles is a great way to iron out the kinks in an operating model. Google docs collaboration features helped us work through and collate feedback along the way.

Compromise is necessary for aspirational principles

With significant size, scale and legacy it will be hard to always respect all of your principles all of the time. What to compromise where and how is key to making sure that trust stays high throughout the change process.

Matrix management reduces alignment

Matrix management structures are really hard to scale effectively. Especially when managers have different goals to the individuals that they are accountable for and those don't match team goals.

Alignment increases impact

To have the most impact build teams of aligned individuals with a shared goal they helped create and believe they can achieve. Then help them to reduce dependencies and remove impediments.

Build collaborative processes

The best external structures to the teams are informal ones that are representative, meet regularly to collaborate on issues and maintain standards. However you need a leader to drive this in order for it to continue over a longer term. If you can create collaborative process where people learn together, regularly talk about issues and are then given time to address them change can be a groundswell rather than trying to push water uphill.

Aligned autonomy takes time

Scaling teams to become high performing through aligned autonomy takes time. You need to let teams test their boundaries to enable them to get comfortable with where their constraints lie within their autonomy. This process is best facilitated through regular structured communication with stakeholders.

Consistency can have a high cost

The larger the number of teams the greater the overhead to enable consistency. One of the biggest challenges with my engineering background was to understand this. People scale in very different ways than code. The right balance between standardisation and customisation is key. This is changes over time and should be inspected and adapted as you move go.

Structured external communication is key

Teams should be able to define and improve their own processes as long as they communicate in a consistent way with stakeholders. The way I think about this is much like a contract for an API define the interactions with the outside world first and as long as they are respected it doesn't matter if the internal workings are different per team.

Metrics are hard

Finding the right metric for a team to focus on also takes time and iteration. Teams need to be given the appropriate time to go deep into the problem space to figure this out. This often requires engineering effort and a good reliable data platform.

Balancing metrics are key

Commercial goals need to be balanced with quality, performance and security measures that the whole team believe in. This takes time and a level of maturity to get right.

Accountability

If multiple people are accountable then no one is accountable.

Hire smart people

Make sure that you hire smarter leaders than yourself, co-create a vision, collaborate on a goal, give regular feedback to enable them to have the maximum impact.

Centre of excellence

A centre of excellence is a great way to ensure that areas of quality but limited resources are utilised and share knowledge effectively. We have moved to a model where they draw up a contract with teams based on a menu of their services. This ensures that expectations are clear and met. This has really helped as we have moved away from having dedicated resources in teams such as agile coaching and testing.

Full Stack

Full stack is a mindset that really can enable teams to have more impact but is good to both train as well as hire. Generalists can also have negative impact on quality so it is a hard balance to get right.

Effective communication underpins everything

Communicate, communicate, communicate. Communication scales exponentially I do not.

Autonomy, Master & Purpose

Autonomy, mastery and purpose are key to setting teams and individuals up for success. However getting the balance right with autonomy takes time, teams and individuals take longer to find their purpose and you need to think about the balance of experience levels in teams with regards to mastery.

The key learning from the last two and a half years is how hard significant change can be at an organisation level. Timing is key which means approaches need to evolve over time as there is no right answer to a lot of questions. Persistence, belief and drive are key to reach the tipping point where change itself becomes a natural part of the process.

Consistent revenue growth is more important than fast audience growth

This is the statement which has changed the most in the last fifteen months. When The Tab raised in 2015 it was to drive audience growth via an American expansion. The revenue following that growth was a secondary concern. Having since been involved in a funding round I can say that this attitude has changed. Clear, consistent and diversified revenues are a huge part of owning your destiny or being able to raise additional funds to grow.

The free growth years are over

As most people have noted the free growth years for media are over. Some say Facebook is capping the amount of traffic that it is sending to publishers. They are definitely focused on optimising the type of content displayed to each of their users. Also the more you are locked into their ecosystem the less attractive you are for investment.

However there are still opportunities for growth

As Facebook rolls out new initiatives there are still windows of opportunities for growth. Ripped video content for example has been very effectively growing page likes. However you usually need to skate close to the line to make this happen and quickly backfill with proper content once your knuckles get rapped.

Facebook Instant Articles provide a decent revenue stream

Considering all the alternatives early indications are that Facebook Instant Articles are a decent revenue stream. With programmatic on mobile still 12-18 months away from where it is on desktop and users demanding a fast experience (as well as Facebook using speed as a Newsfeed ranking factor) this is good news. With auto-fill rates above 95% and decent CPM's so far for The Tab's audience this does seem like a good play until other mobile units catch up.

Throwing resources at growth does not build a sustainable business

What is the key driver of your business and revenue? Find that and focus on it relentlessly. It is much easier to build new things rather than fix core issues. Especially when flush with venture capital. Doing more does nothing but move you further away from what you actually need to fix.

Focus is hard when you are unsure of what you are being measured on

What matters most? Uniques, revenue, page views, stories published or cost per story? There are so many things you can be measured on at times it can be bewildering. Bottom line is that you need a clear repeatable, scalable and cost effective growth model including a plan for profitability. Everything else should drop out of that.

There is still a large appetite for great original content

So much of the media industry has been focused on the most efficient way to repackage content which means that everyone is fighting over the same eyeballs. However there are still a lot of amazing original stories out there you just have to have an efficient way to find them and deliver to a relevant audience.

Raising money is a full time job

The time and effort that goes into raising funds is a full time job. It can take many months, hundreds of meetings and due diligence to close out a round. It is a non-linear process as well with many people you meet along the way introducing you to more people to talk to.

VCs with aligned portfolios

When selecting a venture capitalist one that has a decent sized aligned portfolio can provide you access to learning and introductions that others can't. In such a competitive industry these insights can be gold dust and definitely worth considering as you search for investment.

Negative aspects of raising

Not knowing if you are going to have enough money to continue as your runway is short and terms sheets are close but not signed can really dampen momentum. It takes very strong belief to be able to continue to drive forward in the face of such uncertainty. The volume of feedback you get from pitches can be overwhelming at times. It can be easy for this to become a distraction which can really hurt focus and belief.

Positive aspects of raising

Putting your company on the line and distilling everything you down to a slide deck and presenting it hundreds of times can help sharpen your focus. Some of the feedback you receive throughout this process will be valuable, the trick is to figure out which bits. The pressure which comes from this process either makes you or breaks you, at times it can do both in a single day.

More money, more problems

One of the biggest losses with fast growth by VC money can be the loss of creativity that comes from constraint. Getting the balance right between hitting short term targets and long term issues is hard.

I am very proud of what we achieved with The Tab and I have every confidence in the team going on to achieve great things. These lessons should give me a great foundation for my next challenge.

The Tab Team

The Tab's Team analytics platform was created with a main goal of providing actionable analytics data in a competitive framework for individuals and teams. The product aimed to get authors excited about how individual efforts had contributed to their teams overall success. What you see in red is an authors impact on the team performance. Authors are given a clear overview of how each of their recent stories has performed. At the highest level an overview of the top performers is presented as a leaderboard broken down by country and then team, author and story allowing multiple layers of competition and learning across our network.

Previous to this project data access was restricted to a small set of users with a Google Analytics login and a high level understanding of custom filters. WordPress also had some stats but only at a team level and they were reasonably hidden within the dashboard.

Take the data to the people

The initial launch was a web based front end, the next step was an upgrade of the notification system to be able to send detailed data summaries via email. Distributing data via email increased users to Team by 200% with close to 100% of our active authors logging in monthly. These emails have been able to maintain a 65% open rate and over 10% click through rate after being sent thousands of times.

Data everywhere

Analytics are a natural form of gamification and authors were constantly posting screenshots in Facebook groups or Snapchats. To further enable view ability and use stats were embedded on every page of the site for logged in users. The goal was to show as much data as possible within the visual interface that was being used daily. A completely open approach to numbers allowed authors to learn from each others success, there are no restrictions in who can see what data once someone is logged in.

Real time notifications

Team has been running successfully since November 2015 delivering a clear path to re-engagement and a core part of The Tab's retention strategy. Since then notifications have been extended to be able to deliver realtime readers stats as well as historical page views. These notifications are delivered by an iOS app The Tab + as well as Facebook Messenger. Push notifications enabled The Tab + to have a very similar level of use and retention to Team. The number of interactions with users has increased as the data is broken down into multiple smaller messages and sent as soon as thresholds are reached. This helped authors to stay excited about their existing story for longer and the more they enjoy process the more likely they are to repeat it.

Actionable analytics for editorial decision making

In a bid to take the data to where editors gather a Slack bot was built to enable access to detailed Google and Facebook data just by pasting in a url. Slack channels were created where real time publish and stats updates allowed editors to make decisions about what to promote across our network. Previously editors had to log into multiple systems to gather this information and spend a lot of time filtering. Decisions are now able to be made and discussed in near realtime.

Big Screen Dashboards

With authors being served with data in a competitively actionable analytics framework editors within the office were the next to be targetted. Utilising Geckoboard, custom API's and Google Sheets The Tab designed big screen dashboards to drive competition and growth centrally.

It was a lot of effort to get the data in the right place and then expose it in creative ways to drive learning and performance but now competitively actionable analytics has permeated into The Tab's culture there is no going back.

The geeky bit

Millions of rows of data a day are processed using Node.js, Amazon Simple Queue Service, Elastic Beanstalk Workers and MySQL running on RDS. The data is cleaned and then transformed into day, month and year increments as well as aggregated for users and teams. This preprocessing allows very fast recall of any data set. Amazon API gateway using a Node.js Lambda based server less API layer is used for data retrieval. This handles a lot of the standard API paradigms like security and caching which kept the focus on data and user experience. The front end was built in React.js utilising Chart.js for graphing.

The team

Big up to Richard Coombes for helping with the backend magic, Matteo Gildone for helping with the frontend magic, Serge Bondarevsky for design and Charlie Gardner-Hill for the dashboards. My focus was on the product management, data collection, data transformation and building the API layer.

The benefits of app users are huge in terms of their engagement and propensity to return daily. Being able to target users via push notifications is a great way to create a trigger habitual consumption of your content. These work best when targeted, relevant and contextual or are quickly turned off.

However these advantages can come at a high cost in terms of acquiring new users. We ran extensive internal banner advertising campaigns on metro.co.uk with limited success. Banners are not the greatest advertising medium, especially mobile banners with a call to action. I think we were naive to think that people would want to install an app whist browsing the web.

You can get much better results from retargeting campaigns based on device and previous visit to your site. Especially if retargeting via Facebook app installs ads. However effective they are at acquiring customers without a clear ROI they were a cost we could not bear.

Metro already had multiple apps for newspaper based consumption. Being yet another app in a constellation hurts when you are the new kid on the block. Visibility from search in the Google Play store also really hurt us due Metro being a very common name. The beta nature of our approach also ruled out using our contacts for App Store promotion.

While developing I released as often as I could. Pushing a daily alpha build and using it on different devices than what I was using for development. This could be quickly rolled out to production every other day once bugs were fixed. The fact it only took hours to get these in the hands of users due to automation was a great advantage of Android.

I ended up with one mobile, one seven inch for coding and one mobile, one seven inch for testing releases. I also managed to get a good group of beta testers in our Google+ Community for feedback.

Google analytics is amazing for tracking performance and especially bugs. Crashes only get sent through to the Play store if people submit but show up in Google Analytics regardless. Due to the limits of our testing (I was the developer and tester) this feedback was invaluable. The sheer number of devices on offer also made a release and fix a necessary approach.

I put a lot of effort into tracking all of the actions that people took within the app. This was a really useful dataset in helping make some big product decisions. I would recommend this approach to all development. It wasn't a large amount of effort to setup either.

Push notifications are very important not just to be clicked on but as a visual reminder of your app on the phone. We utilised Parse from Facebook to get this capability setup without much engineering effort. However these had to be manually sent due to engineering constraints.

The best growth hack we did was run a competition asking for feedback via a Google Doc. For some reason everyone who left us feedback also left us a positive review with really helped us avoid the cold start syndrome with reviews.

We had a small vocal minority which reached out to us from App Store reviews. Quickly fixing their issues and responding helped us turn a few reviews around and got some great product feedback. My biggest amazement is the amount of people that never upgraded to a new version even though the one they were on was buggy.

With all of our efforts we only managed to get a few hundred daily users. They were and probably still are a very loyal bunch but not enough to maintain ongoing development. Plus I left for a new challenge so that didnt help momentum.

My gut however is that the real issue with apps based around single brand content consumption is that people's habits have changed. Users want to dip in and out of news from their social feeds and friends updates. Without large marketing budgets due to the lack of significant revenue uplift getting on users home pages is a struggle and building enough difference/providing enough content is a struggle to keep them engaged.

A software container is a lightweight virtualisation technology that abstracts away the complexity of the operating system and simply exposes ports to the host it runs on. You can run containers on most operating systems and platforms including all major PAAS providers. Keeping the complexity within the container means that host systems can focus on scaling and management. You also get a high level of consistency allowing you to ship containers across different servers or platforms with ease. Essentially building once, saving the image and then pulling on to each of your environments for testing and then deployment.

Utilising the micro service instance per container pattern is a great way to manage a set of services which have the following benefits. Including increased scalability by changing the number of container instances. A great level of encapsulation which means that all services can be stopped started and deployed in the same way. You are able to limit the CPU and memory at a service level. Finally containers are much faster to work with than fully fledged virtual machines and simpler to ship around to platforms for deployment. Amazon for example has built in support via their Container Service as well as Elastic Beanstalk. I have used Ansible for deployment as it has really nice Docker wrapper which makes starting, stoping, pushing and pulling images between servers only a couple of lines of code.

One of the things that you need to watch out for is container reuse as they can get quite large. Base images are a way to minimise this and promote reuse and allow you to be able to control the underlying approach to multiple repositories without code duplication. Each step within the build of the Docker container is cached so only stages that are changed need to be pushed when changed. So be careful around the order that you run scripts leaving the stages that change till last. DockerHub is like a GitHub repository for built images and makes the pushing and pulling images require minimal infrastructure and learning. You can pay for private repositories in the same way that GitHub allows you to, or you can setup your own one if you are that way inclined.

Running Docker containers locally on a Mac is pretty straight forward with Boot2Docker which spins up a local Vagrant box which has the docker daemon running on it and allows you to easily test, build, push and query the main docker repo. Kitematic was also recently acquired by Docker as an alternative to people who are adverse to the command line. There are also a large set of officially maintained base container images for running Node, Jenkins and WordPress amongst many others. You need to understand some patterns around how best to ensure that data is persisted as if you stop a Docker container you loose the data within it. However data only containers are a way around this and a pattern that allows you to persist containers without losing the data or binding it too closely to the underlying operating system.

Microservices allow us to choose the right tool for the job and Docker containers abstract some of the complexity of this approach. Utilising base images promotes code reuse and the Dockerfiles themselves are checked in with the projects so any one who pulls the project can see how it is built which is a huge bonus. Most downsides like persistence and the size of containers have strategies to minimise their impact.

I would be very interested to hear your thoughts and experiences with microservices and containerisation.

Vagrant wrapped VirtualBox for local virtualisation

Vagrant is a handy wrapper for VirtualBox which allows you to programmatically setup underlying OS, mapped drives and define provisioning scripts. It also configures your local networking so you can see it as localhost. This can be checked into to your source control for ease of distribution as well as inbuilt change control.

Ansible for provisioning

Ansible is a configuration management tool that connects via SSH and programatically runs scripts for you. You use a simple language to define what each step will do and has default helpers for a lot of different tools such as Docker. The great thing about Ansible is that you can use it for both provisioning of servers and deployment of code. This is also checked in to a repository for distribution and change control.

Same scripts locally and at all stages

The advantage of using Ansible is that you can utilise very similar scripts for provisioning across every environment. The task based approach allows you to construct subtle differences but increases confidence of consistency. Automation of deployment also helps ensure that from your local environment through each stage that code lives in the same places executed in the same way.

Package, build and tests management

Ensuring that your development machine has the correct versions of packages, build and test tools can be quite a challenge. Especially when working on multiple projects with subtle differences. Having a script that configures these takes away a lot of the upfront setup and ensures you are running the same versions as your continuous integration servers.

It works on my machine

This is a such a common cry from developers and something we are slowly moving away from. As our local servers run a VM which is almost identical to development and production testing locally is much more realistic. It also forces developers to consider that there code will be running on a server. This is an important mindset that helps move away from this issue.

My IDE does that for me

The main pushback I have encountered is from people using a fully fledged IDE which contains a web server. I think that these two approaches can work in tandem which a push to your local server before checking code in as an extra step. I have also put an IDE within the server for an even higher level of consistency.

Cost and productivity boosts

The quicker that you can find a problem the easier and cheaper it is to solve. The lack of context switching required as you are awaiting feedback from test and regression is also a real bonus. No longer can it be the operations teams responsibility to push code to production, developers should comprehend the impact that their code makes on all environments and local virtualisation really helps this mindset. Being able to switch between different environments with a simple vagrant up is definitely a future I want to be part of.

Set the right long term goal to give context to all decisions

In Jack's case this was the ability for him to go to a normal school without any assistance. This context provided assistance to making harder short term decisions.

Complex questions rarely have a 100% answer

Doctors are rarely able to give you a 100% answer to complex questions. I have come to appreciate this way of thinking as it avoids setting you up for disappointment. This reflects the reality of the unknown and changable nature of environments.

The more people involved the harder consistent communication becomes

Communication between all of the parties involved in 24/7 care is a constant challenge. This can be helped by writing things down, putting them on walls and doing as much as possible face to face.

Start from the worst case scenario and work backward

Planning for the worst ensures you really think about all options. This is a great mind hack to be happier with outcomes that aren't the best case scenario.

Capture as much information as possible

Over longer periods of time it is essential to write down decisions and observations so anyone can revisit the context and data around decisions if they weren't involved in them at the time.

Establishing baselines and thresholds help autonomous decisioning

Every baby is unique and collecting data is a great way to understand their current state compared to their history. Once you have established a baseline it is easier to empower people to act if thresholds are broken. Overall population baselines are also useful over a longer term view.

Monitoring should be visual and constant

All monitors should be highly visible and when something deviates from the established baseline then they should alarm. Alarms should have clear levels between their various states.

Daily stand ups are essential

A daily conversation with all of the people that are going to be involved in the care of the child are essential. This coupled with data enables distributed decision making. Face to face conversation ensures everyone gets the chance to contribute.

Choosing the right option when many are available is difficult

There is a decent amount of trial and error in solving complex problems. There are standard approaches which give you options for the next step but only by trying and measuring will you actually find out how effective they are.

A clear path of escalation is essential

Knowing who to ask if you are blocked or have an emergency is essential. This coupled with having access to people with greater levels of experience can really help move things forward. The last 8 months have been an incredible journey and I am unbelievably grateful for everyone who has helped us along the way. This process has broadened my approach, understanding and mindset for managing complex projects. I am thankful to the systems that have enabled Jack to have the smile that now warms my heart on a daily basis.

"Scarcity captures the mind," explain Mullainathan and Shafir. It promotes tunnel vision, helping us focus on the crisis at hand but making us "less insightful, less forward-thinking, less controlled". Wise long-term decisions and willpower require cognitive resources. Poverty (the book's core example) leaves far less of those resources at our disposal.

The editorial process is driven by risk aversion, due to a lack of ability to change a printed product after the deadline. Banks of subs check and recheck work and a single person runs each section as well as an overall editor. These create multiple bottlenecks which adds significant overhead, coupled with low levels of autonomy which increases the queuing further. Most of the paper remains a work in progress until the last possible minute. Many of the systems that enable editorial processes are very old and not built for the current world we live in. Complex to change and expensive to run, this is the final piece holding back progress. Change requires running multiple concurrent systems. Each of these is tightly coupled with other complex systems and risk aversion is high. Cost to achieve this change is very high in monetary terms, training and complexity. Complex process coupled with complex systems and a reduced cognitive capacity for anything outside of their deadline has been holding back editorial progress for years. I believe this is one of the reasons why newer entrants without this legacy have fared so well. They don't have any of these previous constraints. Change is possible but usually underestimated due to the multiple layers of complexity and need to continue the existing process whilst building the new one.

1: Ensure everyone has a clear vision of the end goal

This needs to be concise, measurable and most importantly achievable. A strong reason behind why that goal was chosen will help motivation. Everyone should be clear on what they can do to affect the goal. This should be the main job of leadership.

2: Use small cross functional, self organising teams

Should have as much autonomy as possible in how they affect their goal. This is key to enabling faster decision making which increases learning velocity. Proximity is the best hack to maximise face to face communication. This is the most effective way to ensure a common understanding.

3: Timing is everything

The biggest challenge is building the right product/feature, at the right time, using the right technology. The biggest waste is building things that people don't want or need now.

4: Focus on figuring out the next releasable step that takes you closer to your goal

Ensure it is small, releasable and gets you both closer to your goal and provides valuable feedback.

5: Prototypes are a great way to improve early and ongoing feedback

Paper/whiteboards are a great place to start as they allow the quickest iteration. Later prototypes are most effective when viewed in the medium they will be delivered in e.g. In browser/device.

6: Project plans should as high level as possible

They are great for making a high level view of major deliverables visible. If they constantly need updating they are too granular.

7: UX/Design should be 2-4 weeks ahead of development

Designing and building prototypes with a goal of getting as much feedback as possible before development begins.

8: What is designed and what is built are two separate things

Both should inform the other but there is no master due to constraints on both sides.

9: Just in time is the best approach to detailed planning

Any earlier can be wasteful due to risk of new data from earlier releases or prototypes. Pair up on ticket writing using face to face communication and attach any prototypes/mockups/wireframes.

10: Less is more with process once you have a mature team

An agile journey must start somewhere and a fixed process like Scrum/Kanban is a great place to begin. However as the team and process matures your aim should be to reduce this to the minimum possible for your environment.

11: Centralised communication is best done outside of email

Slack/Trello are great examples of products that allow a participatory conversation without the cognitive overload of email.

12: Evolutionary architectural approach works best, complexity shows where work is needed

The simpler you start the quicker you can get real feedback. Avoiding over architecture allows you to combat scaling issues when required, usually by following established patterns. This avoids adding unnecessary complexity early on which can seriously hamper your ability to learn fast.

13: Micro services are a great pattern for a service based architecture

The ability to pick up, modify and release a service with complete confidence that it won't impact anything else helps you move faster. They also allow you to prototype new technologies in a production environment. Focus on automation up front to minimise overheads of running multiple services.

14: Focus

The less we build the better we build what we do. Cognitive overload of working on multiple things at a time has a huge impact on quality.

15: Limiting work in progress is the best way to speed up delivery

Queues are very ineffective, the more queues you have and the more items they contain the worse they perform.

16: Product feedback should happen on a regular basis and have all stakeholders attend

Feedback should be constructive, open and honest. As we release everyday we have two demos a week and our Slack channel is constantly open for feedback. This is where work is prioritised, discussed and tweaked.

17: Data wins arguments

It's ok to loose a battle based on opinion to come back and win the war with data. Make sure you are measuring the right things, this takes time but is worth the investment. Then look for anomalies or what I call "data smells". Following these to their cause will give you great insight into your product.

18: Innovation needs time and space to happen which initially needs to be forced

Hacks days/afternoons are a great way to kick start this process. Give people the a fixed time with a clear direction and see what they come up with.

19: Beware of the curse of knowledge

Don't get frustrated, embrace the fact that some people aren't as far along the journey as you and help them take those next steps.

20: People should have strong opinions that are weakly held

An opinion is always useful to speed up the ideation phase. Logic around environmental constraints should shape the final decision.

21: Embrace your constraints

Each environment has a unique set of constraints. Use these to aid quick decision making. This should give you time to focus on what you need to change long term to be most effective.

During my time at Metro the only constant was change. We were able to embrace this and use it to our advantage. Iterative learning based approaches helped us maintain consistent growth. Delaying every decision until we had the best data possible kept our failures small and valuable. Building great products is about forming a team around an achievable goal and iterating based on the best feedback available at each stage.

Evolution of the Metro.co.uk homepage over the past four years

׋›

׋›

׋›

׋›

׋›

׋›

׋›

Score

((Tweets + Facebook Interactions) * Social Multiplier) + Views

Trending

Score Now - Score 30 Mins Ago

Social

Tweets + Facebook Interactions

Date

Date descending

Coefficient

((((Tweets + Facebook Interactions) * Social Multiplier) + Views + Editorial Boost) * Hour Coefficient) + Tag Boost

We also pass in the following to modify the results depending on the channel, e.g. news, that the data sits within.

Hours to return

e.g. From 24-336 since publishing

Filter Subcategories

e.g. Football,Oddballs,Weird,Food

Boost Tag

e.g. arsenal-fc

Social Multiplier

e.g. 10

Content Type

e.g. Blog or Post

Remove Tags

e.g. tottenham-hotspur-fc

Coefficient

e.g. 0-4 * 3, 4-12 * 1. 13-24 * 0.5, 25-48 0.3

The coefficient sort now takes input from the articles that the editors place at the top of each of the channel pages of the site. This editorial boost allowed us to keep everything feeling much fresher until the data catches up. We have also built our own real time page view tracking system with Node and Redis to get around the time lag in Omniture of 30-60 mins. We recently centralised all of the settings so that they are easy to view and change. I focused on optimising the cluster of content returned and timeframes to retrieve within to get them working with publishing patterns per channel. The ability to cluster content of similar channels has helped ensure we offer a wider variety of content at different stages of the user's journey. Using different sort methods coupled with this clustering has reduced duplication. The design has also helped by using different image ratios and colours for different sections of that page that may contain the same content. We have standardised the bottom of all pages to be the same. This means if we are able to improve performance then it is felt across the entire site. The last addition was the ability to boost by tag. This has enabled the article based algorithm to be much more relevant. At this level of granularity we decided context is much more important than freshness. Moving the tag boost outside of the coefficient enabled this to be clustered at the top but we limit this to the five most recent related articles. Our API is able to deliver everything that the front end needs for rendering including title, image and excerpt. This has also enabled us to use this data in multiple places such as the native Tablet and Phone Editions and the glance journalism experiment Metro10 and MetroVision. Our feeds finally go through an additional layer that allow us to add sponsored content at fixed positions for traffic driving purposes. The great part of all of this is that the maths is still very simple and can be explained to anyone who is interested. Having a set of values to tweak per channel has enabled us to have enough options to slice the data for use in multiple contexts. It has taken 12 months and a full redesign to really see this come to life but I hope it will be a part of Metro for years to come.

Home Page

Metro Homepage Placements

Article Page

Metro Article Page Placement My talk at the WordPress VIP Big Media Meetup on this.

1: Good editorial content will deliver more growth than any product based approach

With a single well written/planned/timed story able to deliver millions of page views and course through the veins of social networks for weeks this should be the number one focus.

2: Good UX turns the dial more than any product hacks

The better the experience of product and content the more likely people are to visit your site, share your content and form habits around its consumption.

3: The closer to the main content area of the page the more related the content should be

Our data has shown that the closer to the article body or top of channel pages the better contextually related content perfoms. Once you are below these areas users are more open to a wider set of content to continue their journey.

4: Where content is placed on the page is almost as important as the content that is placed there

Our testing revealed content placement is almost as important as content selection (as long as it is relevant and recent). This is one of the reasons we have moved to an algorithmic approach for large areas of the site.

• Nothing beats the value of an editorially selected contextual link within the article body
• The area just after article delivers a lot of value as users have finished reading and can be easily tempted into something else
• Sidebars aren't shown on mobile and banner blindness often turns them off for desktop users so they are not an area we focus on

5: Fill dead space with content, people like to scroll it's the natural behaviour of the web

Our newsfeed delivers over 10% of the page views of our site, this is pretty impressive considering it used to be blank space at the bottom of every article and channel page.

6: Don't mess with the natural way that the web works

We tried and failed with this during our swipe phase. 5-7% of users delivered 20% of our page views but that didn't increase their overall time on site. However it complicated everything we built hampering our ability to learn fast. It also didn't quite fit into commercial or editorial strategies. This frustration/learning was what inspired the algorithm and scroll based newsfeed you now see.

7: Algorithms are great but need help from humans to perform at their best

Simple algorithms are a great way to optimise editorial workflows especially around content positioning. However these are only as good as the data behind them. Often you have to wait for this to be gathered before acting on it. Using editorial intuition is a great way to shortcut this process. Especially if you can make it run off existing priorities then process change isn't required to participate.

8: Whatever Google/Facebook ask you to do just do it

They deliver so much of your traffic don't question, just do what they recommend.

9: Feed the beast

Google and Facebook are always hungry for quality content. Gaining momentum requires constant feeding. They both have overall scores for domain as well as article urls so focusing on keeping this high means a better chance to gain and then maintain momentum.

10: Think of every page as a funnel, you loose users as they scroll but the lower they get the more open to their next engagement they become

The higher up the page something is placed the more people will see it. However the lower down the page someone is the more open they are to being tempted by some more content, advertising or interactions (e.g. poll vote, comments)

11: A mobile first approach is a great way to approach product prioritisation

Most of our traffic comes from mobile rather than desktop so it is logical to prioritise. This has formed a major part of our growth strategy.

12: Goals need to be concise, measurable and focus on why

The more people understand the goal and are able to affect it the more powerful it is. A goal that contains a why will always beat a goal that just contains a what.

13: Product specific performance should be broken down to actions per daily active users for comparison

This gives a much better overview of actual performance. Allows you to take out traffic fluctuations, just make sure you have enough data.

14: A week seems to be the minimum amount of data required to see if a feature has worked

Due to fluctuations in traffic and browsing habits. Also good to look at monthly and quarterly trends over longer periods as quite often they exhibit patterns that aren't found at lower levels. It was asking questions around unexpected trends/data that helped teach me most around product growth.

15: Distribute weekly reports to show trends and give your stakeholders an overview of how the product is performing

Have these scheduled to your team and stakeholders via email. Also very useful if you break something when fixing something else. Great safety net to minimise impact and spot any unexpected growth.

16: Any new feature needs to be taken in context of how it fits in the editorial work flow. The closer it is to the existing process the more likely it will be adopted.

The best way to change a habit is build off an existing trigger. New features that leverage existing habits will get much higher adoption than building new habits/process.

17: Consider the users current journey and their emotional state in all features

Segmenting users based on mindset is a great way to understand data. e.g. Social browsers are likely on a multi site journey in a chromed browser on a mobile device. So they are only looking for a single story from your site so optimise for that. No point in worrying about pages/visit focus on getting more return visits via a social follow.

18: When coming from social users are often looking to enhance their social status

Our top share buttons get clicked on 4 times more than our bottom share buttons. Social proof around number of others already shared also promotes more sharing.

19: When coming from search users are usually in a topic based mindset

More likely to click on related, in article links and masthead channel links. Continue to deliver great content around a niche to form habits. Particularly useful around passion centres e.g. Premier League clubs.

20: It's better to have 100 amazing tag pages that look and feel like a destination than 10,000 that feel like they were made for Google

Quality trumps quantity every time, Google knows if you users are clicking through.

21: People click on headlines 4x more than they click images

This is why A/B testing headlines is a great idea. It is the single piece of the editorial process that can have the biggest impact on growth. We also have SEO and socially optimised headlines to ensure we cater to both needs. These are the principles that I have applied to the product development of metro.co.uk over the past two years. The key takeaway is that constant experimentation is a great way to unlock growth if your environment supports it. The hard part is achieving that without adding too much complexity. Complexity inhibits your ability to learn and learning is central to any successful product growth strategy. Building a set of guidelines has enabled us to move faster and helped foster our continued growth. One for the future.

Micro interactions help drive habitual use

We don't have a lot of data on this yet but there seems to be a correlation between micro interactions such as poll votes and habitual use. My theory is that by engaging different parts of the brain you become more memorable. These simple actions form the basis of new habits around content consumption. I think this is a major opportunity for future growth.

׋›

׋›

׋›

׋›

׋›

׋›

׋›

Metro 10 was born out of a desire to experiment with native mobile news reading experiences that solved a different problem to our already fully responsive website. We had an algorithm that allowed customisation and decided to use this as our data source. The concept of restricting the volume of content would help us differentiate from our infinite web experience. We also wanted our users to engage with each article fully before moving on and not just get into a state of skimming which we had seen with infinite lists online.

Our goal was to minimise the time to launch so we decided a hybrid approach would help this. Utilise the native elements for caching and navigation but a web view for the article body. As our CMS stores all of the HTML and other markup in a single blob stripping this out was too much work for the first iteration of an idea that was unproven. Our initial ideas were based on simple mobile interaction patterns. Swipe to dismiss, interaction based personalisation and full bleed images as our tablet edition had experimented successfully with this. We decided on an Android first approach as we had Java skills in house and we also wanted to iterate quickly once it had launched. The turnaround in the Play store would allow us to do this.

I hadn't written any code in four years and getting back in the saddle was a rewarding experience. Leaning a new language and platform at the same time took a little time but the tools and ecosystem are pretty mature. Initially I got out my little black Moleskin notebook and started sketching. Putting something down on paper helped me to start to visualise flow and basic function. I am no UX designer but I find sketching a really valuable first step in any design process.

Once we had a basic idea we jumped into a rapid prototyping phase. This involved a lot of copying and pasting from Stack Overflow and some working from home to get some focus to get the bare bones of an app together. It was buggy as hell but being able to show people on a phone and see there reaction was priceless. I then involved Matt our UX designer and he started to iterate on the design. We were lucky that the concept was simple and allowed very quick iterations. The design quickly morphed from a list that looked very web based to a full bleed image that felt more mobile first.

It was at this stage I got a few more of the team involved. When rapid prototyping it was good to be solo and really iterate quickly but once we were locked on an idea we needed stability and to start on a refactoring process to move towards a proper architecture. I believe in emergent design where you don't over architect things up front but as you learn more of the systems and domain build what is most appropriate. This can save a lot of time as you only build what is necessary and refactor once you are sure of its value.

The next phase of development to our first alpha was frustrating at times but also very rewarding. We kept the requirements process very fluid as things would change on a daily basis. This was a challenge but having written user stories which were quickly out of date and fought with automation of testing we decided keeping things simple was the best approach. Guerrilla usability testing has been key to shaping the UX. As the initial idea was pretty out there putting it in real peoples hands in the real world was priceless. We quickly learnt that people liked the limited number of stories, enjoyed the full screen design but having to dismiss every story Snapchat style wasn't something that they felt comfortable with. The paradox of choice was blinding them. With 10 possible stories to read and only one visible at a time and only one chance to read it they couldn't decide whether it was interesting enough.

We decided to strip the user experience back to its most basic and only give people 10 stories at at time that they can scroll through left and right. With this approach people were quickly able to grasp the concept and understand the navigation. Calling it Metro10 ensured that we kept to our goal of just the 10 most popular stories right now. Throughout this process there have been a lot of tension between what constitutes our MVP. Due to a fixed timeframe we had to make some compromises around fluidity of design. The biggest was to leave all interactions via gestures rather than fluid animations. The level of complexity rose exponentially with these.

We were able to get validation of our ideas without them. Learning if the concept works is the most important and all of the underlying data, service and business layer need to work no matter what the front end looks like. Overall it has been a great experience in taking a concept from paper to reality in less than two months. There have been lots of zigs and zags along the way but that is what makes this job interesting.

MVP

׋›

׋›

׋›

׋›

׋›

׋›

׋›

׋›

׋›

׋›

׋›

Later iteration

׋›

׋›

׋›

׋›

׋›

Changing the mindset of the business to make mobile a first class citizen was a big challenge. However once they had agreed on a mobile first paradigm we were able to tackle the hard questions up front and once this allowed the development team to focus on how best to approach the responsive elements.

Image resizing was a key part of keeping our file sizes low and having a fast perceptive render. We also dropped our custom font on mobile due to render speeds. An adopted an adaptive approach that only delivered the HTML necessary for the page to render and prioritised mobile for speed reasons.

Mobile first ensured we spent our time focussing on the main content that the user had come to read. We went for the simplest option of not including the sidebar of content on mobile as it would only add page weight. The volume of clicks in this area also justified it not being included. Users have a sidebar blindness and unless it is the main navigational feature will rarely get used as the brain turns it off to focus on reading the main content area.

We are currently going through a new redesign process which will give a new lick of paint to what we achieved two years ago but the principles remain the same.

The below is a presentation delivered at the Google HTML5 Roundtable about Metro's journey into responsive design

1 - To begin the pen is mightier than the keyboard

Starting with a paper and a pen is a great way to iterate quickly through ideas. Sketches are one of the easiest forms of communication at this stage. There lofi form means you are usually more comfortable with changing them.

2 - Keep the concept simple

We went through many stages of adding complexity to Metro10 but we always came back to the simplest implementation. With the lack of space on a mobile screen and wide audience we intended to go for this was what kept redefining our approach.

3 - Rapid Prototyping is your friend

We managed to get a version on a phone within 3 days that had our initial concept on it. This helped us put it in front of a much wider set of people very early in the process. It didn't run off feeds, ran out memory quickly and had random image sizes but was key to getting a wider set of people giving us feedback on our concept.

4 - Guerrilla usability testing is a great way to get early feedback

As we had a prototype early I just started going up to people and giving it to them without any instructions. Watching their frustrations and listening to the questions they asked provided amazing feedback to help shape our product.

5 - Image resource management on Android is not straight forward

We probably spent close to half the time on the app stopping out of memory errors due to cropping and displaying of images. By making sure you manually clear the copy of the image that is attached to the image view we managed to fix this.

6 - Do as much processing as you can on the server

We ended up using the WordPress image resizer to resize all of our images to the specific screen size before they got to the device. This helped speed up processing and ensured we only downloaded the image size we needed.

7 - 7" tablets are actually treated as large phones

Android treats 7" tablets as very large phones rather than small tablets. This can mean your app needs some adjustment to look good on the larger screen.

8 - Google Play Store Alpha and Beta Communities are the best use for Google+

Google allow you to set up a community on Google+ which will give pre-release access to members. This was a great way to get feedback and manage the roll out of the app. It really made this part of the process a pleasure.

9 - Unit testing and front end automation for repeatable testing

We used both JUnit tests and front end automation to quickly get feedback on every change. We moved away from the Android unit tests due to speed issues with the emulator. We managed to integrate those and the front end framework with our CI environment though the management of emulators again provided some interesting challenges.

10 - There is no substitute for using real devices in testing

Emulators provided a great way for us to get started quickly but using a mouse to control features proved tricky. We also saw different behaviour on emulators and real devices over the time period. The best way to learn something is by doing it, by starting with a set of constraints and a focus on simplicity and I am very happy with the results we got with Metro10. We have now passed 500 downloads, are maintaining over 100 daily active users and still have over a four star rating.

Popular = ((Tweets + Comments + Likes + Shares) * 25) + Views

We decided to give the social signals a significant multiplier as a share action gave a much stronger signal of how much the content was liked than a page view. Based on this thought process we started with a very high multiplier of around 50 but reduced this to 25 after looking at the results over a period of time. This was calculated every half hour after fresh data was collected. The main issue with this approach was the frequency of change within the top stories. So we changed ordering to be based on the rate of change between the previously calculated score. We called this trending and it gave a really interesting snapshot of what was popular on the site and this improved the frequency of change.

Trending = Popular Score (Now) - Popular Score (30 minutes ago)

When we redesigned the site this sat right below the top five stories on the homepage. When we check the stats we were surprised that the number of clicks were almost identical to the clicks on the top module that were editorially selected and also had very large clickable images. Clearly people were interested in what other everyone else was interested in. This data also proliferated across our sidebars and into our Tablet and Phone Editions via an API. It updated 24/7 and required no manual intervention so no matter what time you visited the site it displayed a mix of fresh popular content. Based on the numbers we decided to see if this could be extended to run more of the site.

One of Metro's competitive advantages is that it runs a very lean digital operation. We want people to spend their time finding and writing great content rather than worrying about placement. A relatively small volume of our traffic goes to our channel pages (e.g. News) or looks at sidebars due to the volume of mobile traffic we pull in. So we decided to experiment further with this concept and get it to run the rest of the homepage. We had placed a timebased feed at the bottom of the homepage and had been very surprised with the number of interactions. Just being time based wouldn't work for the homepage as we needed a measure of popularity, freshness and time. I sat down last December and started prototyping approaches to take. We had to give new stories a chance to be featured before they had as much data. We also didn't want one huge story to be at the top for too long. After much deliberation/time spent running around Hyde Park I decided I would use a coefficient based on story age to boost posts in the early stages and penalise them once they had been live long enough to have an unfair advantage.

(Views + (Social * 25)) * Hours Since Published Coefficient

Hours Since Publish

Coefficient

0

25

1

15

2

5

3

3

4

1

4-8

0.7

9-12

0.3

13-24

0.05

25-36

0.02

We had also been talking about how we could distribute content around the site better. Especially at the bottom of articles as this is where the majority of our traffic comes. This is when we placed the same timebased feed from the homepage underneath every article in what used to be empty space. We were amazed once again at the level of interaction this received and decided to optimise the data behind it.

The main idea was rather than restricting the stream to just articles from the specific sub-category (e.g. Sport/Football) you were in, we could use the highest level category and boost the subcategory you were in. Eg see all of Sport but have all Football stories closest to the top. This would give other very popular sports stories a place as well as aiding circulation around the site.

(Current Channel Boost + Views + (Social * 10)) * Hours Since Published Coefficient

I then added an option to allow you to pass in the channels that a user visited the most and using the same logic to prioritise their content near the top. Again not excluding content but rather ordering it in a way that was most appropriate for the person viewing. This is going to form the basis of our new Android App with it just displaying the top ten stories based on your browsing habits.

(Users Channels Boost + Views + (Social * 10)) * Hours Since Published Coefficient

The final twist was to use the data that we had collected on the backend to style the front-end. Our News Feed now changes its imagesize depending on if a story is trending or has been flagged by editorial. This give prominence to the popular stories as the algorithm pushes them further down the list and keeps the styling changing 24/7. As we use data from existing editorial placements it also means that there is no extra work to manage. We have worked on optimising the placement on this and we are now getting over 3 million News Feed lazy loads being requested weekly and over 500,000 extra page views. Considering this area of the site used to be blank this is a tidy return on investment. The latest enhancement is to grey out stories you have read to give an even bigger prominence to stories you have yet to consume.

The other large jumps in interactions were triggered by:

• Changing the height on when additional content was pulled in
• Increasing the size of images for trending stories
• Putting comments behind a tab and making timeline the default view

The hardest part of this process was getting all the data in the right place from the APIs in the first place. Once this was present and we had a nice API around this the rest of this process was a iterative effort between editorial and development. There are a lot of moving parts but the benefits of having a site that reacts to users behaviours and works 24/7 is something that we feel is hugely worthwhile.

Over the past 12 months Metro has been on an amazing growth curve. Some of it is being in the right place at the right time for algorithm changes but a lot of it was planning and then execution of a growth (hacking) strategy.

Hack 1: Responsive Design

Monthly Unique Visitors from Mobile (Jan'12 - Feb'14)

We decided a responsive design would be the best way to capitalise on the explosive growth in mobile. A nine month redesign process culminated in Metro going responsive on 7th Dec '12. We immediately saw growth from social referrals with Twitter's almost doubling over night. The other benefit of responsive sites is that there is only one URL. As this is the key used to store ranking information in search and social algorithms you don't want this split between multiple domains like m dot.

The key takeaway is that if you give people a great experience on all devices then they are much more likely to read, share and return to your content.

Weekly Visits from Twitter (Oct'12 - Feb'13)

Hack 2: Focus all development efforts on growth

WordPress.com Stats for Metro (Dec'12 - Feb'14)

We also migrated to the hosted platform as a service provided by Automattic on vip.wordpress.com. This enabled all of our costs and resources to be focused on growth as they didn't have to worry about caching, servers or anything that didn't improve experience for our readers or editorial users. The amazing thing about the platform is that it is a flat fee. This means that although our traffic has grown 350% year on year our costs have not changed. The depth of their out of the box features plus ecosystem of plugins ensured that we did't have to worry about commodity features such as SEO, site maps and editorial workflows as someone else had built and open sourced an approach.

Hack 3: Open up content creation to anyone

Metro Blogs Monthly Unique Users (Dec'12 - Feb'14)

The other great thing about the WordPress platform is that it enabled us to allow bloggers to contribute content directly into our core CMS. This started out as a feature for Metro employees but grew to encompass a much wider set of sources. Club Metro is now the most prominent and now contributes over 1M unique visitors a month. Having blogs content on the same domain placed amongst the rest of Metro's content ensured we leveraged our existing algorithmic rankings. A single editorial workflow also helped keep the overheads low. The added bonus that most bloggers were already using WordPress and can write from anywhere helped us secure some top talent.

Club Metro Article

Hack 4: Facebook page as a major marketing channel

173,000 Facebook Page likes Feb '13 to 562,000 Facebook Page Likes Feb '14

Social referrals were another large contributor to growth. We focused on growing our Facebook page "Likes" as a content marketing strategy and this was very successful. We had always been careful to only send a small amount of our best posts out a day on our Facebook page. This had a solid base of users and growing this was a key goal for the last year. We employed many strategies including competitions with like gates and boosting posts. Competitions were effective in bringing in users but there were a lot of repeat entries. Varying the prize helped to minimise this and our email based CRM platform really helped to drive entries. The most cost effective way we found was boosting posts as friends of people who already liked Metro were shown a great piece of our content. They were then much more likely to then go on to like the page. As they were similar to people who already liked Metro they were very receptive and continued to engage with our content.

Metro Facebook Page

Hack 5: Made to share content

Weekly Social Visits to Metro (Jan'13 - Feb'14)

Ensuring that not just the content put out on the Facebook page was made to share also really helped grow our social referrals. When the content is written they set a success measure of 100 shares on an article and then set performance targets on achieving that on 25% of our content. This has allowed some interesting conversations around the areas of the site where that was less prevalent. More than anything it is a very easy test for all of the content creators to know what they should focus on. If it won't hit that bar then find something else. A key growth hack we developed was the ability to show different headlines for social and search so we didn't impact our search traffic.

SEO and Social Headlines on Made to Share Content

Hack 6: Made to share UX

Share and follow functionality on Metro.co.uk after the Made to Share focus.

The other side of social was increasing the number of people sharing from the site. The development team had a made to share focus where we introduced much larger, clearer social buttons and reduced the number of clicks it took to share. This with the addition of a sticky sharing bar that floats on desktop has seen a large increase in the number of direct shares from our site. This seems to have also affected the amount of people copying and pasting links from the site from the subtle reinforcement due constantly present share cues. It would also seem that Facebook take direct site shares as a strong signal in their algorithm as we have continued to see growth from social.

Hack 7: If something feels wrong don't give up on it

Referral from Natural Search (Google Fixed, July '12)

After the redesign only 20% of our stories indexed in Google News had our pictures next to them. We spent months experimenting on different options before we finally managed to ask Google the right question in July '13 so they could fix it for us. It turned out that they were still using the Webmaster Tools account we had been using before the migration which pointed to our old domain that included www. Not only did this change help our referrals from Google News but it gave us a major kick in all search referrals. It would have been much easier to give up on this earlier but relentlessly focusing on this until we solved it really paid dividends.

Hack 8: Let technology automate repetition

Metro Development Releases to Production (Nov'12 - Feb'14)

Automation of all of our development and test processes allowed us to release 4.5 times on average every day (apart from Fridays) for the past 12 months. This enabled short feedback cycles and decision making to happen at a much faster pace. We have five different environments that are used for testing before we push code live. This kept errors to a minimum and kept the feedback flowing. This environment of automated front end tests and frameworks was a major investment but has continued to pay dividends.

Hack 9: Ensure proximity of key people who are focusing on a goal

Single Goal: 700,000 Average Daily Mobile Visitors in September 2013

A single goal of growth allowed us to work together cross functionally and a focus on data and numbers ensured that feedback was alway digestible. Content, social and tech sitting together and working together enabled the good ideas to come to the top quicker and equally the bad ones get ignored. Equally focusing on data helped take emotion out of decision making which enabled data to win arguments. This sped up innovation and focus. In most cases we have done less but done what we have done better to achieve growth.

Hack 10: Get out of the way

Voticle: Are you a true Brit?

Once people are working cross functionally together towards a goal then get out of the way and let them get on with it. In the past two months we have relaxed our process and now the content creators are working directly with the developers on new article formats to continue our growth. Out of this we have developed five new ways of displaying content from quizzes to lists and beyond.

Quizicle: How much of a Londoner are you.

Conclusion

None of the above would have been possible without the adoption of a lean mindset and the approach of build, measure, learn, iterate. It has been an amazing 12 months of growth at Metro and an great feeling to be part of a team that came up with and then executed a plan which delivered these results.

It was a pleasure to work with the below as well as many others on this journey.

• Martin Ashplant @mashplant
• Jack Dearlove @jackdearlove
• Richard Moynihan @richjm
• Gareth Noon @andfinally
• Matteo Gildone @domsmasher
• Matt Humphreys @mhumph
• Hoque Ali @hoqueali
• Paul Kevan @pkevan
• Carlos Esteves @kalej
• Dominic Rowell @domrowell
• Jamie Walters @jamiewaltz

SELECT CONCAT('Redirect 301 ','/slugtoreplace/', post_name, ' ', guid)
  FROM wpdb.wp_posts
 WHERE post_type = 'post'
   AND post_status = 'publish';

You can then post the output from that script into your .htaccess file and everything will redirect properly.

-- update site
UPDATE wpdb.wp_site
   SET domain = 'local.blog'
 WHERE id = 1;

SELECT *
  FROM wpdb.wp_site;

-- update blog details
UPDATE wpdb.wp_blogs
   SET domain = 'local.blog'
 WHERE blog_id = 1;

SELECT *
  FROM wpdb.wp_blogs;

-- update site meta
UPDATE wpdb.wp_sitemeta
   SET meta_value = 'http://local.blog'
 WHERE meta_key = 'siteurl';

UPDATE wpdb.wp_sitemeta
   SET meta_value = '127.0.0.1'
 WHERE meta_key = 'dm_ipaddress';

SELECT *
  FROM wpdb.wp_sitemeta
 WHERE meta_key in ('siteurl','dm_ipaddress');

-- update options
UPDATE wpdb.wp_options
   SET option_value = 'http://local.blog'
 WHERE option_name in ('siteurl','home');

SELECT *
  FROM wpdb.wp_options
 WHERE option_name in ('siteurl','home');

-- update posts
UPDATE wpdb.wp_posts
   SET guid = REPLACE(guid,'blog.david-jensen.com','local.blog');

I have gone through each of the tables above that need updating, you could probably get away without updating all of these but I figure that as it is a script you might has well do it properly. I have also included the select statements afterwards to ensure that you can see the value has changed.

It takes time

It is hard to get communication right within a team, even when they are sitting next to each other. Often it can take months rather than weeks to build up the right cadence of communication and understanding that are required to build trust. This process is even harder when you are also in the midst of creating a new company as it is only one of the many things that require focus. Luckily there are templates that help you hack these processes to make them happen quicker.

Agile mindset

Working with an agile mindset encourages regular feedback and can provide templates for the flow of information and cadence of communication. It also gives you the regular opportunity to tweak your process to ensure that it fits your environment in the form of retrospectives. Proximity is another clear hack and the closer that people sit the easiest it is to get them to communicate.

Standups

I have found that daily stand ups are a great way to hack communication within teams. I do daily stand ups with development teams and at least weekly stand ups with the business and stakeholders. These are best done around the agile wall and include new features as well as an overview of roadmaps. Regular feedback in the form of small regular conversations beats larger infrequent conversations as they allow for regular course correction. This helps prevent waste that comes from working on things that are not the most important for the business.

Consistency of language

The other benefit of regular communication is that both sides have the ability to learn the others language. Having a commonly understood language takes time and reduces the waste caused by miscommunication. Trust is much easier to lose than it is to build and in the majority of cases it is a miscommunication that leads to erosion of trust.

Find common ground

When people are under pressure to deliver it can be easy to blame something you don't understand. There are many different cultures that make up an organisation and how you build a software system isn't the same as how you create budgets. In order for cultures to understand each other you need to establish regular communication over common ground.

Have a clear vision

Common ground should be a clear vision based around a set of hard but achievable goals that everyone can affect. Clear goals enable decision making to be decentralised which helps to avoid one person blocking progress. Maintaining flow is essential to the creation of momentum which forms the baseline of high performance. Making your vision and current goals very visible should allow your team to see how the work that they are doing is affecting the key business metrics.

Conclusion

When teams are communicating well within a clear framework of success it allows everyone to focus on delivering value. The ability to influence business goals with regular feedback creates an ideal environment for high productivity. Communication is an often over looked discipline when forming teams and especially companies. It is the foundation of common understanding and this builds trust that will allow you take your performance from good to great.

First install varnish.

sudo yum install varnish

Make sure that it comes back on startup.

chkconfig varnish on

Now edit the following file.

sudo vim /etc/sysconfig/varnish

The below is what I ended up with in my file thanks to this. The main difference from the original is the change of the VARNISH_STORAGE_TYPE to malloc which means that it is stored in memory rather than disk. I chose 128MB as my blog is pretty small and I wanted to have a bit of room to spare.

# Varnish Users
VARNISH_RUN_USER=varnish
VARNISH_RUN_GROUP=varnish

# Should we start varnishd at boot?  Set to "yes" to enable.
START=yes

# Maximum number of open files (for ulimit -n)
# NFILES=131072

# Locked shared memory (for ulimit -l)
# Default log size is 82MB + header
MEMLOCK=82000

# Maximum number of threads (for ulimit -u)
NPROCS="unlimited"

# Maximum size of corefile (for ulimit -c). Default in Fedora is 0
DAEMON_COREFILE_LIMIT="unlimited"
RELOAD_VCL=1

# Should probably change this
VARNISH_VCL_CONF=/etc/varnish/default.vcl

# Not setting VARNISH_LISTEN_ADDRESS makes Varnish listen on all IPs on this box
# (Both IPv4 and IPv6 if available). Set manually to override this.
# VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80

# Telnet admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082

# Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret

# The minimum number of worker threads to start
VARNISH_MIN_THREADS=50

# The Maximum number of worker threads to start
VARNISH_MAX_THREADS=5000

# Idle timeout for worker threads
VARNISH_THREAD_TIMEOUT=120

# Best option is malloc if you can. malloc will make use of swap space smartly if
# you have it and need it.
VARNISH_STORAGE_TYPE=malloc

# Cache file size: in bytes, optionally using k / M / G / T suffix,
# or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=128M

VARNISH_STORAGE="${VARNISH_STORAGE_TYPE},${VARNISH_STORAGE_SIZE}"

# Default TTL used when the backend does not specify one
VARNISH_TTL=60

# DAEMON_OPTS is used by the init script.  If you add or remove options, make
# sure you update this section, too.
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -t ${VARNISH_TTL} \
             -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
             -u ${VARNISH_RUN_USER} -g ${VARNISH_RUN_GROUP} \
             -S ${VARNISH_SECRET_FILE} \
             -s ${VARNISH_STORAGE}"

Then you need to edit your /etc/varnish/default.vcl the below is what I ended up with which allows you to see the WordPress Admin Bar but caches most other bits. It also takes invalidation requests from the Varnish WordPress Plugin which is the easiest way to keep your content fresh.

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

acl purge {
  "localhost";
}

sub vcl_recv {
  if (req.request == "BAN") {
    if(!client.ip ~ purge) {
      error 405 "Not allowed.";
    }
    ban("req.url ~ "+req.url+" && req.http.host == "+req.http.host);
    error 200 "Banned.";
  }

  if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
    return (pipe);
  }

  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

  # don't cache authenticated sessions
  if (req.http.Cookie && req.http.Cookie ~ "(wordpress_|PHPSESSID)") {
      return(pass);
  }

  # don't cache ajax requests
  if(req.http.X-Requested-With == "XMLHttpRequest" || req.url ~ "nocache" || req.url ~ "(control.php|wp-comments-post.php|wp-login.php|bb-login.php|bb-reset-password.php|register.php)") {
      return (pass);
   }

  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (pass);
  }

  remove req.http.cookie;
  return (lookup);
}

sub vcl_fetch {
  if (beresp.status => 400) {
    set beresp.ttl = 0m;
    return(hit_for_pass);
  }

  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (hit_for_pass);
  }

  set beresp.ttl = 24h;
  return (deliver);
}

You then need to update your Apache to listen to port 8080 as Varnish will be listening on port 80.

sudo vim /etc/httpd/conf/httpd.conf

Update Listen 80 to Listen 8080 and then restart Apache and Varnish and then you can run varnishstat to see the details of how your cache is performing.

sudo service httpd restart
sudo service varnish restart
varnishstat

I then installed the Varnish WordPress Plugin and then went to Dashboard -> Settings -> WP-Varnish and updated the following fields.

• Varnish Administration IP Address
• Varnish Administration Port
• Varnish Administration Secret

The secret is a GUID which is stored in the below.

sudo vim /etc/varnish/secret

These are some really useful Varnish commands to see how your cache is performing or check out the docs for full details.

• varnishtop: like top but for varnish.
• varnishhist: histogram of request time
• varnishstat: cache hits, resource consumption and many other data points
• varnishlog: log of all requests made to the cache

You should then have a super charged server where most of the files are coming from memory rather than disk. The below sites were helpful along this journey.

Install Varnish HTTP Accelerator with WordPress Speed up WordPress with Apache and Varnish WordPress Varnish Cache Config / VCL

“You must be the change you wish to see in the world.” Mahatma Gandhi

Making change a habit is a powerful way of not just creating but maintaining an agile mindset. In the fast paced world that we occupy an agile mindset has come to be valued over many other attributes of successful companies. Agile is a word that is often used without a true understanding of the mindset that underpins its values.

Mindsets

A mindset is "a set of assumptions, methods, or notations held by one or more people or groups of people that is so established that it creates a powerful incentive within these people or groups to continue to adopt or accept prior behaviors, choices, or tools". Another way of looking at that is a set of habits which an individual/group perform similar outcomes to. Agile has its origins in software development with the Agile Manifesto which was signed in 2001 and outlines the mindset needed to create successful software. The group that signed the Agile Manifesto stated that they value:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

How your brain works

In order to better understand how you create the set of habits that form a mindset it helps to understand how the brain works. There are two distinct systems in your brain used for thinking as described by Daniel Kahneman in his book "Thinking Fast, Thinking Slow".

• System 1: Fast, automatic, frequent, emotional, stereotypic, subconscious
• System 2: Slow, effortful, infrequent, logical, calculating, conscious

System 1 is where you spend most of your time and could be described as the autopilot part of your brain. Basically this allows you to do complex things without large amounts of energy and effort. System 2 is what you use when you encounter a problem that System 1 doesn't recognise and this requires a different deeper type of thought.

Most people spend the majority of their time using the System 1 part of their brain. However like most things System 2 becomes more effective the more you utilise it. If you can take something from System 2 and engrain it in System 1 then you are then able to spend the energy you save on solving new problems. It also becomes the default behaviour for how you respond to this type of situation.

Habits

Another way to describe this type behaviour is a habit, defined as "A settled or regular tendency or practice, esp. one that is hard to give up" or simply something which has been engrained in System 1. They can be created and/or modified and a great description of this process can be found in "The Power of Habit" a book by Charles Duhig. He defines a habit of consisting of the following:

• A trigger
• A routine (what you do to respond to the trigger)
• A reward

The easiest way to change a habit is to understand what the trigger and reward are and to replace the routine in the middle with a newer improved version. There are many triggers that exist in the software development lifecycle such as being stuck on a problem or lack of detail in requirements definition. When you start out most people have different routines for dealing with these type of problems. The ideal place to start to discuss these triggers is in a retrospective. In this environment it is great to understand the differing approaches and choose the most effective one for the group to adopt. The reward for this is the great feeling of shipping quality code on a regular basis.

Change a habit

I have found that to change a habit it is always best to start small and find a habit that will be reasonably straightforward to improve. This allows people to understand the process and see the benefits of the approach. I have also found that small regular conversations are much more effective for changing habits than large infrequent conversations. It takes a while for the habit to form and change and being subtly reminded on a regular basis is great way to reinforce the correct behaviour.

In order to create a mindset for a team you need to gradually change their individual habits to be consistent. This is not an easy or quick task but one that delivers an amazing amount of long term benefit to both the individuals and the company/team that they work for. This is probably the single largest factor in turning people into high performing teams due to the efficiency gains and consistency of outputs. If they have made a habit of hacking habits then they are constantly improving and helping each other to do the same.

Conclusion

It has taken me a few years to create a habit of change at Metro and these are some of the practical things that I did to help that journey. Initially I set a very clear high level goal of releasing software every day. This required a lot of changes and I wanted to ensure people knew what success looked like. I then setup regular retrospectives to create a change dialog. I then gave everyone my word that I would be a willing participant of this journey into change and set a clear commitment to remove impediments. This mainly involved getting the business and stakeholders to change alongside our journey. Once this was setup we started small, started changing and are iterating as there is no such thing as perfect process.

Another reason I like it is that it is very non-invasive caching technique, I know that WP-Super-Cache and the way that it generates static files is probably a better fit for my blog as the load isn't usually very concurrent. However that sounded quite straight forward and I like a command line challenge plus I have a few side projects that this might come in handy for. You also need to get this setup if you are ever deploying anything to WordPress.com which we utilise for our hosting at Metro.co.uk.

As I am running a network, I don't want other users to have to worry about caching and its quirks. The nice thing about Batcache is that it doesn't cache anything when you have cookies from the domain so anyone who is writing can see their changes immediately. I think this and its simplicity is probably the key advantage over some other methods of caching.

The other thing is that if you have root access to your server then it is reasonably straight forward and I thought that I would share how I did this here in case anyone else is interested.

First install all of the libraries that you need to run memcache and access it via PHP, it took me a few times to get this right so there might be a few libraries which are extra below.

sudo yum install memcached
sudo yum install php-pecl-memcache
sudo yum install zlib-devel
sudo yum install php-pear
sudo yum install php-devel
sudo yum install libmemcached-devel

This is the compiler I used for building

sudo yum install gcc

This will build the libraries that PHP needs to be able to access Memcache

sudo pecl install memcached

Now make sure that memcached will start on startup

sudo chkconfig memcached on

I also enabled memcache session handling but haven't turned it on.

Enable memcache session handler support? [yes] : yes

I didn't do as it suggested below as every time I did I got errors saying that it had already been registered.

You should add "extension=memcached.so" to php.ini

I had a look at the config and tried to lock it down to localhost only but it kept on throwing errors when I did so I left it, as all of my ports are locked down via the Amazon Firewall this shouldn't be an issue.

sudo vim /etc/sysconfig/memcached

Now made sure that it would come back on startup.

sudo chkconfig memcached on

Now I followed these instructions for Batcache installation once I had downloaded it.

Upload advanced-cache.php to the /wp-content/ directory

Add this line the top of wp-config.php to activate Batcache:

define('WP_CACHE', true);

Tweak the options near the top of advanced-cache.php, I set mine to cache for 60 minutes after being hit twice in 10 minutes.

var $max_age =  3600; // Expire batcache items aged this many seconds (zero to disable batcache)
var $seconds =    600; // ...in this many seconds (zero to ignore this and use batcache immediately)

Optional Upload batcache.php to the /wp-content/plugins/ directory. As I ran Multisite I put this in the /wp-content/mu-plugins so it would be there for all of the sites in my network.

We have already installed the PECL memcached extension so you just need to add Ryan's Memcached backend 2.0 to wp-content.

Test by reloading a page in your browser several times and then viewing the source once you have cleared any cookies from the domain you are hitting. Just above the closing tag in the head you should see some Batcache stats.

generated 524 seconds ago
generated in 0.374 seconds
served from batcache in 0.002 seconds
expires in 3076 seconds

Now feel free to tweak the settings around cache time to a period where you will reduce the load. Remember this is full page caching and you shouldn't see it if you are logged in.

SEO Impact

The best part of installing Batcache has been the impact on SEO, below are my crawl stats from Google and you can see that the average time spent downloading a page flatlines around the beginning of December after installing Batcache. The really interesting thing is that the pages crawled per day went up at the same time. I think this is because Google now thinks that that server can handle a much greater load so sends it its bots on a more regular basis. Can't say that this has had a massive impact on my SEO entries but it is a really good sign of the impact that you can have with a proper caching strategy.

Super fast and simple caching, if its good enough for WordPress.com its good enough for me. The below articles helped me along this journey.

• How To Install and Use Memcache on Ubuntu 12.04
• Faster WordPress: Multisite, nginx, and Batcache
• Batcache for WordPress

Wendy Lea the CEO of Get Satisfaction gave a presentation at the DMGT Technology Summer School on "The voice of the customer". There were some great points on how they have infused passion for their customers into a customer driven development process. As their business is based completely on capturing customer feedback it was great to see how they have infused their company mission throughout everything that they do. I have collated the main points I took from the talk below.

• "Listen -> Engage -> Iterate" is their mantra for development
• "Advocate -> Discover ->  Buy" is a common loop for new customers
• You need to create and curate a community for product development (Facebook, Twitter handle)
• The Cluetrain Manifesto - "The connectedness of the Web is transforming what's inside and outside your business — your market and your employees."
• Create systems of engagement mashing in your external and external systems
• Conversations created by real people, in their natural language are very powerful and discoverable
• Bidirectional conversation can help that flow
• Join the conversation:
  • Embrace customer conversation
  • Open these conversations
  • Collaborate
  • Apply actionable insights

Embracing the above thoughts into a customer driven development process is something that really got me thinking on how we might be able to apply this at Metro. Apart from the old letters page newspapers traditionally are more of a one way dialogue. I have been thinking of ways that we could apply some of the above and simple feedback forms are something we are going to begin trialling soon as we make further changes to Metro.co.uk.

I had the pleasure of spending some time at the DMGT Technology Summer School with Bill Raduchel. He had a long and interesting career before joining DMGT as a technology advisor to the board. I managed to capture some of his advice below and a couple of quotes which resonated well with me.

"Heroism is an act of stupidity that you happen to survive." Bill Raduchel

• Open source is the only way to get the best developers as they can learn portable skills
• Proprietary systems are struggling
• John Chambers CEO Cisco: We compete against market transitions not companies
• You must get people loosely coupled but tightly aligned
• You must have a vision and be able to sell it, you must communicate it in a way that all your employees can understand
• Asking the right questions is never interference
• You must set the boundaries and then get out of the way
• Architecture is to empower your team, not constrain it
• Commitment before success is the mentality that separates people in a startup from people in a corporation

"Mindset is a value that has become a habit." - Bill Raduchel

• It's the business model, stupid - first place to start when a company isn't working
• Most technical decisions are actually business decision
• It's not just about money, it's also opportunity cost
• Support the lunatic fringe
• Understand the assumptions so you can quickly adjust when things don't work out and they don't
• Innovate around what is core for you and outsource the rest
• You are building for the platform of tomorrow, not of today
• You must understand your customers ecosystem and fit in effortlessly
• If you can't win then recognise early and deal with it
• Think backwards from your customers
• Narrow the customer set to provide 100% solution
• Understand the ecosystem and the business model
• Build 95% solution and pay 5% is Google solution
• 60% of VC's don't return on their capital

I think that the lack of established templates to follow such as SCRUM has made this a harder process as there isn't a template to start from. I also think that in order to get it right the metrics need to be driven from the business, so you are dependent on them being engaged. The toolset that helps you move to this type of approach is also lacking and in the end we had to build/hack our own.

Our journey began by the business setting one key goal backed by a clear metric that the whole business could understand. This was easier said than done and were definitely parts of the business that feel left out by this singular approach. However just having one key metric in the beginning is very important as it aids focus. Metro's was to have an average of 700,000 daily mobile visitors in September 2013.

The next phase was to have a look at all the data that makes up the key metric and to break it down further to show each area of the business what they could effect. Being as open and honest about where the numbers have come from at this stage is essential. There will definitely be questions about how the decision was made especially if it is a large goal and being open at this stage is the best way to combat that (even if it it is a vanity/stretch target).

As the largest part of the goal was going to come from metro.co.uk and that partly my teams responsibility we started by looking at our web analytics (Omniture) to see what data we were already collecting. We also looked to the content and search teams to see what metrics they were tracking as only by working together would we be successful. Common goals should bring people together to talk about frustrations/improvements, this alignment should provide major benefits. However it still surprises me in large businesses how little people actually talk.

I think part of this problem is the fact that so many parts of the business have their own language. Development as much as anyone contributes to this as it takes quite a lot of time to be able to converse with most developers. Marketing, Sales and Content all also have their own ways of working and language which can be just as daunting for developers. Getting the translation right between these different groups so they can talk enough of the same language to communicate effectively is a major business benefit. I think this is one of the reasons that startups are able to iterate faster as they have to talk the same language as you have a limited set of people doing a lot of of different things but all speaking the same language.

Having had and looked at all the existing metrics we decided that there wasn't actually enough specific product based feedback mechanisms to measure the success of our work. So to understand the product more we had to build our own measurement/tracking framework on top of Omniture. This took some work to get right and then took even longer to get the tracking across our entire site. However once we were able to see the results of each of our improvements in both click perspective and overall impact on the wider numbers a really important feedback mechanism was created.

Measuring the impact was only the first part however we needed to get the data into a format that everyone could see and understand. This was to be a common theme throughout this process you go from having no data to having too much data that doesn't make a whole lot of sense. Having dedicated resources and time to create dashboards that compiled this data into meaning was the next step on this journey. Then we created regular meetings that talked around this data and tried to put meaning around it. Product Performance was born and we scheduled it in for every two weeks. We get all of the product owners and stakeholders that are involved in product development into a room and talk through performance of the previous two weeks. We also do a product based standup twice a week where all stakeholders stand in front of the Kanban board and discuss what is being done and about to drop into the process. Lots of small regular conversations definitely provide more value and leverage for change than large infrequent meetings.

We also got large televisions that monitored real time traffic and put them up in visible locations so everyone could see what was going on and feel the excitement when a large story took off. Real time data is the next step in this process and the faster you can see the data the faster you can react to it. The next step after having everything measured was to begin A/B testing. This step again proved to be a much harder than we initially thought. As our site is news based and constantly changing we found we couldn't really use a Javascript based framework like Optimisely as things changes too much. Plus the cost was very prohibitive for a site of our size. So we ended up building a simple framework that extended our measurement in Omniture.

The other key learning was as we release multiple changes everyday what do you test and what do you just measure. We got to a place that if you were working on an incremental improvement then A/B test otherwise just ensure that everything is measured. We were surprised as well that sometimes the A/B results were 50:50 but when we made then change anyway it had a longer term affect on user behaviour that was positive and not captured during the testing.

We also had some issues with our Google News Sitemap and indexation. This was out most experimented feature over months. We ended up creating a dedicated swim lane on our Kansan board just for this. We only ran one test at a time and then iterated to the next one. I think for harder more complex options running just one test at a time really helped focus in the results of each before moving on.

Running too many tests at the same time proved to be confusing and I ended up making one of the team the chief growth hacker and he became the go to person for what and how to run tests. Now that we are in a place with a nice testing framework and approach it was time to see if it would work towards a more major site redesign. We needed to break this big design down into testable chunks that would prove our hypothesis that people would interact more by scrolling than swiping. Creating stories that were just for learning felt like a natural evolution from a lean mindset.

Conclusion

It has been a really interesting nine months and we have managed to hit a goal that seemed completely unachievable only 12 months ago. I think we managed to create a measurement mindset where everyone asks what we are measuring before starting development. This added to the power of everyone pointing in the same direction, having a process that allows constant improvement and promotes communication has produced some amazing results. You should give it a go.

• Start with one key metric
• Break metric down to provide relevance
• Measure everything
• Create dashboards to distill data
• Make the data visible to everyone
• Regular meetings (short and often) to create common language
• Try running some A/B tests
• Keep changes small and regular
• Communicate, communicate, communicate
• Enjoy your successes

Further Reading

This blog post was picked up by The Media Briefing who further interviewed myself and Jamie Walters for a slightly different angle on the above. The post How the Metro uses metrics to create flexible and effective digital products is well worth a read.

Monitoring

The first step that I took was setting up alerts in the Amazon EC2 console to send me an email if my CPU stays at 100% for over five minutes. You can do this by clicking on your server in the console and then hitting the "Monitoring" tab and then "Create Alarm". This seemed to be about the right amount of time to stop too many false positives and usually indicated that something was up with the server.

The second set of monitoring I setup was New Relic, we use them extensively at work to monitor our servers and their free version gives you a load of really useful information about how both the server and PHP are performing. Think of it as a version of Google Real Time that allows you to look at the processes and speed of everything going on inside your server and application. They also have a really useful availability monitoring service built in which you can setup to ping your site every few minutes to see if it is accessible and a great iPhone App that sends push notifications when something goes down.

• Server monitor installation: RedHat and CentOS
• PHP agent installation: RedHat and CentOS

Logs

The next thing you need to do is SSH into your server and check your httpd access_log to see what is trying to access your server.

sudo tail -f /etc/httpd/logs/access_log

This will bring up what is currently happening on your server and you can see all of the requests that are coming in. If your site does go down this is the place to check right afterwards to see what the requests were that overloaded it.

Mod Security

I decided to install Mod Security along with the OWASP ruleset to give myself an added extra layer of protection from any issues that might get introduced into WordPress before a patch comes out. I have to say that this involved quite a steep learning curve and due to the amount of technical information that I write in my blog so it took me a long time to modify all of the rules to allow saving of posts amongst other things.

As I am running on the AWS Centos image and it already had the EPEL repository setup so all I needed to do was:

sudo yum install mod_security mod_security_crs
sudo service httpd restart

Once this has installed you should definitely start clicking around your blog and see what is broken, quite likely that actions like saving a post, updating a comments status and updating plugins will now throw 403 errors. In order to allow your site to function need to create a file called whitelist.conf in the following location.

/etc/httpd/modsecurity.d/activated_rules

You then need to populate it with rules that allow you to bypass the specific rules that are triggered when using WordPress. Mod_security isn't aware of your logged in status and assumes every call is external so this throws up quite a few issues. I have included a link to the ruleset that I have built up over the last few months below. There are probably more entries than most people will need but due to the geeky and code based nature of my posts a lot of stuff gets blocked, I have also enabled XMLRPC so apps work as well.

whitelist.conf

Adding your own rules to whitelist.conf

Should you get a 403 error when performing any action on your site you should SSH to your server and via the terminal type in the below.

sudo tail -f /etc/httpd/logs/modsec_audit.log

Then when you retry the action you should see another entry in the log and you need to pick out the data that looks like this.

[id "960035"]

You then just need to add the following line to your whitelist.conf

SecRuleRemoveById 960035

Under the correct Location e.g.

<LocationMatch "/xmlrpc.php">
  SecRuleRemoveById 981173
</LocationMatch>

This can be quite a laborious process and possibly the whole lot is overkill but have to say I have learnt quite a bit in the process. I also added the following to the bottom of my http.conf which bans an IP from attempting to login for five minutes after three failed attempts.

# Stop Brute Force Attack by banning IP
# http://www.frameloss.org/2011/07/29/stopping-brute-force-logins-against-wordpress/

# This has to be global, cannot exist within a directory or location clause
SecAction phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},initcol:user=%{REMOTE_ADDR},id:2323

# Setup brute force detection.

# React if block flag has been set.
SecRule user:bf_block "@gt 0" "deny,status:401,log,msg:'ip address blocked for 5 minutes, more than 3 login attempts in 3 minutes.',id:2323"

# Setup Tracking. On a successful login, a 302 redirect is performed, a 200 indicates login failed.
SecRule RESPONSE_STATUS "^302" "phase:5,t:none,nolog,pass,setvar:ip.bf_counter=0,id:2324"
SecRule RESPONSE_STATUS "^200" "phase:5,chain,t:none,nolog,pass,setvar:ip.bf_counter=+1,deprecatevar:ip.bf_counter=1/180,id:2325"
SecRule ip:bf_counter "@gt 3" "t:none,setvar:user.bf_block=1,expirevar:user.bf_block=300,setvar:ip.bf_counter=0"

htaccess

The final part of this is the .htaccess changes that I made to stop people being able to post without a referrer as a lot of bots use this method as they are computer programs rather than real people. I also had two specific User Agents that had targeted me which I block all access to my wp-login page.

# Stop protected folders from being narked. Also helps with spammers
ErrorDocument 401 /401.html

# Stop spam attack logins and comments
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{REQUEST_URI} .(wp-comments-post|wp-login)\.php*
RewriteCond %{HTTP_REFERER} !.*(blog.david-jensen.com|dev.mariamjensen.com|www.cafeontherye.co.uk).* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(Windows\ NT\ 6\.1;\ WOW64\;\ rv\:18\.0\)\ Gecko/20100101\ Firefox/18\.0 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(Parsley\ NT\ 1.0\;\ rv\:1.0\)\ Parsley/1.0.0.0 [OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule .* - [F]
</ifModule>

Here are a few sites that helped the creation of this post.

• Stopping Brute-force Logins Against WordPress
• Lock down WordPress admin login with .htaccess
• WordPress Login Protection With .htacces

Definitely interested in hearing if anyone else has any other good ideas around how to approach this issue or has any tips from me as I am certainly no security expert. However after the uptime for my server has definitely improved over the last while with all of this helping to reduce WordPress hacker attacks on this blog.

In the past year I have run two large successful agile projects that have delivered software using the Scrum methodology to build a minimum viable product (MVP) and then transitioned to Kanban once they have gone live. The first one was the Metro.co.uk responsive redesign and CMS migration which was delivered on time and under budget. Since going live it has been quickly iterated to deliver solid mobile and overall traffic growth for Metro. The second project is the Metro Play Casino which delivered its MVP as part of a large program of work that required us to get a fully regulated offshore gambling business started. This is now ahead of budgeted expectations and we are currently scoping the next phase of development. During both of these projects there have been some ups and downs but overall it has been a really positive experience. I thought I would share some tips that I have collated whilst managing these projects successfully using agile methodologies.

1. Initially just define a few clear high level goals to measure success that allow detail to be filled in at the appropriate time.
2. Start prototyping as early as possible, learning by doing is the most effective way to figure out what to do and also what not to do. Being able to actually use something as early as possible is very important. Even if this means releasing something different than your end goal.
3. Don't break down every story in the beginning and add it up to find out how long the project is going to take, this will just leave you with an unmanageable backlog and most of the stories will be out of date by the time you come to develop them. We have found that the best approach is to get a group of experts in the room and play planning poker for the project as Dan North suggests calling it Blink Estimation.
4. Don't be afraid to pivot your MVP or technology choices if the learning that you gather points in slightly different direction than you first thought. Knowing what you don't want is as important as knowing what you want.
5. Clear and consistent goals facilitate quicker decision making. These enable decisions to be made based on logic and this decisioning can be distributed around the team and post validated. Any single point of decision making is going to really slow things down, especially important in the last couple of iterations.
6. If you are still developing a lot of new functionality in the last sprint be worried. Also leave a little bit at the end to tie it all together as things always pop up that you didn't plan for.
7. A more traditional Project Management layer can really help with external vendors and internal communication. Keep project plans at the highest possible level for visibility and better decision making especially on dependencies.
8. Change your process, change it again. Retrospectives are really key to this process so don't forget to have them after every sprint. Be agile about your process.
9. Quite often order comes from short periods of carefully orchestrated chaos, these can help teams form by giving people common frustrations. Listening to these and then adapting your process to remove them helps the team to feel empowered.
10. Get onto your production environment as soon as possible, no matter how similar other environments are they are never the same and issues will come out at this stage.
11. Get developers and the business talking directly if at all possible. Learning together through structured conversation delivers the best business value. These can be facilitated by a Business Analyst or Product Owner but they should not always be in the middle as the lack of knowledge transfer will harm the project. This is especially true when dealing with multiple stakeholders.
12. Bite off the biggest unknowns first and work on them to make them knowns. The best way to do this is to release something early and get real people to use your product. This doesn't need to be MVP but a Minimum Learning Product that delivers some value to end users.
13. You can assemble a group of people but it takes time and effort to form a team. Setting achievable goals and then ensuring they are delivered with a sprinkling of pressure early is a great way to prepare for the final few sprints.
14. Restrictions bring out peoples creativity.
15. In my opinion design should be emergent as should architecture, initially go with the most established patterns however you need to give the team time to fix things if they turn out to no longer be the right choice. This allows you to focus on actual problems rather than solving ones you only think you have.
16. Automated testing should focus on smoke tests and high level indicators when things are rapidly changing.
17. Automate builds and deployments in phase 0 for maximum payback.
18. Constantly engage the business to make them define the problems they are trying to solve and then work together on the right solution for the project and the business in iterative steps.
19. If you are going to take any risks with the project take those decisions at the latest possible point.
20. Celebrate the team and their successes.
21. Things will change constantly so learn to embrace it and hopefully get to the place where you enjoy it. If you have problems detailed rather than solutions then these can then be adapted as you learn.
22. Simple things you haven't done before are complicated to execute. Do not under estimate... or fall into the planning fallacy. Build complexity in over time.

Conclusion

Managing successful agile projects is something that can deliver serious business benefit if done with the right approach. The constant feedback that is available allows iterative learning and if your environment is setup to handle this will allow you to be very nimble. Quite often agile is an approach amongst a sea of different project management methodologies and I believe that if adapted properly to your environment can bring real benefits to most projects.

• Metro
• Mail Online
• Euromoney
• dmg::media Sprint Suite / Dev Lab
• Landmark
• Evenbase
• Broadbean
• Wowcher

I was fortunate enough to also be able to participate in Haribo fuelled hacking and below is an overview of what we built and how we approached the whole event.

Vrsus: The future of real time voting

Trying to find a group in the initial hustle of the team forming stage was easier than I thought. My initial idea was something based around real time analytics but this quickly morphed into a real time voting tool to decide which the best cats on the internet are. The great thing about the real time approach was a chance for us to use a load of new technologies. So the dream team that we ended up with consisted of the following:

• Mary Morgan (Euromoney Institutional Investor)
• Frédéric Occédat (Mail Online)
• Panagiotis Albanis (Metro)
• George Old (dmg::media)
• Me (Metro)

Once we had a general direction with regards to online voting we had an hour design session around our flip-board to try to conceptualise the flow through our app and what screens and actions would be necessary. This gave us a decent enough understanding of the requirements to put together a technology stack which consisted of the following tiers:

• Amazon EC2 Ubuntu Instance
• Twitter Bootstrap
• Angular.js
• Socket.io
• Express
• Node.js
• Redis

As we wanted to be real-time and socket based most of the stack chose itself and we thought that Redis would provide a simple way of storing the data and retrieving the post via score. At this stage we started to get down to the detail of building the initial flows around being able to submit images for voting on.

I also started work on setting up the EC2 instance so we could easily pull from GitHub and keep the server running. There were a few things that I need to do in order to get the ports forwarded to HTTP and the Node process running in the background for which I used the following:

• Forever - makes Node.js run as a background process
• Rinetd - allows port forwarding on Linux so Node can run on port 80 and sockets on 8080

Next steps were to get all the pages setup and the logo done which took the rest of the evening.

Coming in the next day first thing we sorted was our Kanban board to get everyone working on the different bits that were required. This worked well through out the day as we finished the following screens.

I also managed to procure the domain name vrsus.me and then added the subdomain you.vrsus.me to it to give it a bit of entertainment. The second day disappeared far to fast and I focused on stripping down the homepage to be nice and simple and also on the create page the ability to preview the image as soon as it has been added using sockets. We decided to against our idea of being able to enter any URL and for us to go and scrape the Open Graph or Twitter tags from it to get the images from the page to just entering a URL of an image.

This proved to be a good idea as it took us the rest of the day just to get the basic flow working, due to Redis not having any relational abilities there it took a bit of time to figure out how to store both the complex hash around the upload as well as the score after voting. The ability for the list to change order in real time also came right down to the wire and although there isn't a fancy animation the list view is socket based and will change order when votes come in which I think adds a lot to the application.

It was then time to give a presentation of how we had approached everything and I had managed to knock up the below in Prezi.

We won the 'Most Technically Impressive' category overall.

Future Ideas

If we were to have some more time to develop this application into something more fully fledged I think these are the area's that we would focus on.

• Ability to post any url and vote on it
• Social integrations
• Ability to put sponsored posts in it
• General responsive and cross browser performance
• Improvements to the animation
• Tracking

I think that Vrsus could be utilised across DMGT in many different ways apart from just showing the best cat beards on the internet. Having a real time voting system that people could submit content to that allowed the best to rise to the top based on user interaction would be a valuable asset in my opinion.

If you are interested in having a look at the code you can see it on GitHub and the url you.vrsus.me is still live and will be so for the next month or so.

Video

DMGT Hackathon - Google Campus from Dan Buck Joyce - OFILMS on Vimeo.

Photos

I was asked to give a lightening talk (less than 5 minutes) on a subject and I decided I would give a quick overview of how we have approached building Swipe into metro.co.uk. This seemed to go down quite well though qz.com were on after me and what they have done with their front end is even more impressive. I thought it would be a nice idea to put my slide deck up somewhere for viewing.

UPDATE: Metro have decided to remove swipe from Metro.co.uk for these reasons.

One of the other talks the previous day was by Yuri Victor on why the Washington Post uses WordPress and if you haven't already seem it then it is definitely worth the visit even for the typography alone.

Other great notes from the few days were to use the following for WordPress debugging.

• Debug Bar
• Debug Bar Console
• Debug Bar Cron

Also use Grunt to automate the use of these two libraries for image optimisation.

• jpegoptim
• optipng

Generating fonts to help reduce images/sprites can be a good idea or just use Genericons which have been done for you. Also I found out that script tags on a page are completely blocking.

We use components within our Jira project to categorise tasks by the business area and product owner that they deliver value for. Each component allows you to setup a default owner and this ensures that they are notified when any tickets are added or amended. We created an additional Kanban rapid board specifically for backlog management and added "Backlog", "To Do" and "Scheduled for Development" columns to this. We then enabled swimlanes based on components so each of the individual backlogs can be prioritised separately within these. We also utilise quick filters based on component to allow each product owner to only see their tickets on the rapid board. As the filters are URL based this allows each product owner to have a browser bookmark to take them straight to their backlog.

Setup Swimlanes per Component

Setup Quick Filter per Component

Greenhopper Multiple Backlogs Kanban Rapid Board with Swimlanes and Quick Filters

Getting the business engaged with development process and priorities in Jira is a long term project of mine (as otherwise I am the only person who manages the board). The best way that I have found to do this is hold specific product owner stand ups twice a week where we have a look at the work in progress and discuss what the priorities are for the items about to move into the development stream. We have this around the TV that has our development Kanban rapid board displayed on it and can quickly switch between that and the backlog view. This allows each of the product owners to get a good view of overall priorities and where their requests fit into that. They also see how the process works and have an opportunity to provide feedback. This is complimented by a meeting every other week where we look at how each idea is delivering value based on performance data and then decide whether we do more or less of this based on our overall strategy.

Having multiple Rapid Boards for a more complicated flow that keeps each of the boards simple is one of the best moves that we have made. We are also now using a separate board to tightly manage our testing and release process as we have multiple environments and stages that overcomplicate the main board. Greenhopper has been really good at adding new features and streamlining their existing ones to provide a great way of managing the different stages of the project in a single dashboard.

Metro.co.uk Greenhopper Kanban Rapid Board Dev Board

Background

I have been using the Atlassian stack including Jira and Greenhopper to manage multiple Agile projects using its Scrum templates over the last three years. The great thing about Jira (and at times the most frustrating thing) is that it is pretty much infinitely configurable and Greenhopper gives a great drag and drop interface for Scrum and Kanban setups. Personally I don't have the time to manage both an online system and a paper based post it system so we moved everything into Jira and got a 42" television which is used for displaying boards during standups and as an information radiator during the day.

During the minimum viable product (MVP) phase of the metro.co.uk CMS migration we had a nice simple setup of Greenhopper utilising its Scrum template with a dedicated Business Analyst (BA) who managed the backlog. However he was a project resource and his contract was about to come to an end, we had also shifted from using Scurm to using Kanban but were initially struggling with the flow from the large backlog left over from the project.

Scrum to Kanban

1. In order to release as often as possible, focus on smoke tests and automation of deployments between environments. Usually this requires a pretty complex setup that will take some time to find the optimal process.
2. Use feature branches to allow small changes to be encapsulated and tested on each environment. This can take some getting used to but the pay off is well worth it.
3. Don't under estimate the power of a Dev Ops culture even if you don't have a specific person for this role empowering both developers and testers to have the time to set everything up correctly will save you time and errors in the future.
4. Break things down to the smallest possible developments based on the feedback that they can provide to avoid blockages. This doesn't mean large projects can't be done but should be packaged in multiple small releases.
5. Limit work in progress (WIP) as well as items on the backlog. A large backlog is usually the most inefficient part of this process. No longer can it be the dumping ground for ground for random ideas. If it isn't done within three months I auto delete and see what people complain about.
6. Hold product performance meetings rather than sprint planning. Bring feedback and measurements and invite both business and development resources and let the data win the arguments. Setting the measurements up takes a while but they payoff of everyone looking at the right metrics is worth it. You will also need a solid goal to be aiming at.
7. Much more conversation needed on a regular basis, we now hold product owner stand ups twice a week as talking every other week isn't enough when you are moving that fast.
8. Talk about problems define small solutions to provide learning then iterate.
9. Whiteboards are your best friend. All of the walls in our new office are whiteboards and it really helps quick brainstorms. Also allows you to leave complex problems in visible place to revisit over time until solutions become clear.
10. Don't forget retrospectives.

Conclusion

Now that we have been running with Kanban for over three months it is amazing how much faster the team has become. Breaking things down into small pieces is probably the biggest skill that is needed to keep the process efficient. Also ensuring that the business are engaged throughout the process has taken some time but the benefit is huge. Agility has also increased as now when we find an issue or discover a new key piece of information you are able to act on it straight away rather than waiting until the next planning meeting. If you are interested in reading in more detail about Scrum vs Kanban then I highly recommend reading Scrum vs Kanban by Henrik Kniberg.

The first critical issue that it threw up was to turn KeepAlive On, this means that Apache keeps the session alive to transfer multiples files. This reduces the latency as each new request for the multiple files that makes up a web page don't have the overhead of opening and closing a connection. I found this great overview of the options here which is worth reading if you want more details.

I then made the following changes to my /etc/httpd/conf/httpd.conf

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 50

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 3

The second critical issue was a lack of compression so I needed to turn compression on and then ensure that all of the file types that benefit from compression were enabled with mod_deflate.

#
# AddEncoding allows you to have certain browsers uncompress
# information on the fly. Note: Not all browsers support this.
# Despite the name similarity, the following Add* directives have nothing
# to do with the FancyIndexing customization directives above.
#
AddEncoding x-compress .Z
AddEncoding x-gzip .gz .tgz

<FilesMatch "\.(php|css|html?|xml|txt|js|pl)$">
SetOutputFilter DEFLATE
</FilesMatch>

The next issue that it identified was a lack of browser caching so I updated my .htaccess file to ensure that files that could be cached by the browser would be as recommended in this post.

## EXPIRES CACHING ##

ExpiresActive On
ExpiresByType image/jpg "access 1 year"
ExpiresByType image/jpeg "access 1 year"
ExpiresByType image/gif "access 1 year"
ExpiresByType image/png "access 1 year"
ExpiresByType text/css "access 1 month"
ExpiresByType application/pdf "access 1 month"
ExpiresByType text/x-javascript "access 1 month"
ExpiresByType application/x-shockwave-flash "access 1 month"
ExpiresByType image/x-icon "access 1 year"
ExpiresDefault "access 2 days"

## EXPIRES CACHING ##

Finally having spent some time on the Google PageSpeed website I saw they had developed the mod_pagespeed Apache module which combines a whole load of Apache web server performance best practices into a single install.

wget https://dl-ssl.google.com/dl/linux/direct/mod-pagespeed-stable_current_i386.rpm
sudo rpm -U mod-pagespeed-*.rpm

Finally I reran PageSpeed Insights and got an overall PageSpeed Score of 94 (out of 100) which was great for a couple of hours works to really speed up WordPress Apache.

My story to come

As you have just heard A+ Media’s digital revenue has recently surpassed their newspaper revenue. This rapid evolution was largely driven by a change in organisational thinking. The focus moved from opinion based decision making to a factual based iterative approach. This increased the speed of innovation and decreased the costs and risks of trying new things. The process was a disruptive one to the wider business but it was recognised that it was better to disrupt ourselves than be disrupted. Management started by setting goals that focused on digital revenue across the whole of A+ Media. They then used a data driven model to validate a large number of ideas around new digital revenue opportunities. For each revenue opportunity they formed a customer based hypothesis that they used their considerable client contacts and existing audience to quickly validate. Next decisions were then based on this real world feedback. A+ Media have used this way thinking to foster a culture of constant innovation from iterative learning. The knowledge that data will win any argument over ego was a critical factor in mobilising the whole of A+ Media behind their digital goals. So I guess you are thinking what does this have to do with me on your induction day? Well you will be taught these techniques and they will help you to shape both the future of A+ Media and each of your respective organisations. This process is called Lean Friday’s and there is a dedicated team who organise and facilitate these. They will be in contact soon, as each of you will be involved in at least one of these sessions a quarter. Lean Fridays are our idea validation process where we take a cross functional group of people and get them to validate our ideas around digital revenue opportunities. In a competition at the end of every month validated ideas are chosen for further data driven development. The team that validated the idea is also given the opportunity to continue to be involved. Idea generation and selection is taken care of by a market based approach. Every employee is given the ability to submit an idea in Salesforce for consideration. They are also given a set amount of A+ currency to invest in ideas that they think will provide the most amount of value. Ideas are ranked in Salesforce based on the amount of investment in them. When a successfully validated idea is turned into revenue then a small percentage of profit is given back to both the initial creator and investors of the idea. They are also given additional A+ currency to invest again. An example of this was: “Stephan Fowler a developer from Metro who came up with the idea to create a marketplace for commercial content creation. Internal knowledge and testing validated how to structure this so editorial would be correctly incentivised and clients validated how much and what they would be willing to pay in trail runs. We then quickly developed an internal prototype using open source software. Over time this was refined and then finally released into the wider marketplace. It has since created over £10 million of profit and Stephan received £10,000 for this.” Highly visible examples like this have ensured that everyone is interested in the generation and validation of ideas and put constant innovation at the heart of our culture. Good luck with your ideas and investments if you would like to know any more about the process please don’t hesitate to ask.

Conclusion

Waste is a huge problem for large modern organisations, usually business models are optimised over time to reduce that waste. I believe that by ensuring the right validation of ideas we can reduce the waste that occurs at the earliest stages of new business model generation. Lean Friday's would be an approach that I believe that would work to do business idea validation at scale. I also believe that they would solve one of the other problems that large organisations have which is communication. By bringing a group of people together around a common problem you can quickly learn how other parts of a large business work and gain some insight and knowledge which is very valuable. Technical companies have hack days for this very reason and people like Facebook and Atlassian are already harnessing these approaches to massive business benefit.

Self Organisation

You obviously also need motivated individuals, the right rewards and a clear achievable goal but when I came back in after the deadline you could almost feel the difference. The team had managed to deploy all the code but not quite configure Akamai correctly to deal with cached User Agents for the mobile site. They had self organised to get a far as they did and only an unknown consequence of a design decision stopped them from meeting the target. However the learning from this mistake ensured we made the right choice later in the project and avoided User Agent detection in favour of using widths. The restriction of time, platforms and approach ensured that they had a common frame of reference to approach the problem and organise around.

Emergent Design and Technical Debt

The team had also hacked a lot of things to meet the deadline which they were now all passionately complaining about. I am a big fan of emergent design where you let the pain points of a project show themselves before fixing. This was a great example of that and I then let the team use the rest of the week to clean up the technical debt and you could see that sealed the team's formation. If you let talented passionate people speak their mind together about constraints and then give them time to fix those issues you get what you actually need not what one person views as the best approach. However if you don't give the team time to clear the technical debt you will loose respect and all of the good work. They also wont feel empowered to make their own decisions in the future.

Order comes out of Chaos

Order comes out of chaos rather than overly thought order descending into chaos. Wrong assumptions made before problems were fully comprehended have derailed many other projects I have been involved in. Leaving all decisions to the last possible point when the maximum amount of learning has been completed is always a good idea. When people have a clear mental model of the problem rather than an out of date one from previous projects that they assume to be valid. As I mentioned earlier you need to insert chaos at the right time in the project as if it is done too early then you can lose more than you gain.

High Performing Team

To see the passion from everyone having made some compromises for a short term goal and their insistence that this could not continue I knew that this group of people was now a team. The learning that they had from self organising around a clear goal and solidarity against the technical debt showed they were well on their way to be a high performing team.

Key Learnings

• Constraints help creativity.
• Clear goals help people self organise.
• Removing a leader for a short period can help self organisation.
• Allowing pain points to surface due to constraints makes people passionate.
• Passionate people perform better if you then empower them by listening to their opinions and fix the technical debt.
• The group needs a certain level of maturity before trying this approach.

As I previously wrote Metro decided to use the WordPress VIP platform for their CMS and front end, this post will detail how we set about the initial part of the project to give it the best chance of success. Initially it was pretty daunting looking forwards 12 months with the sheer scale of the project and diverse set of stakeholders that would judge its success. During this time the development team would also be switching from Java to PHP, have to learn a new platform and completely change the user interface for all of the people who manage and interact with Metro.co.uk on a daily basis. We had a few whiteboard sessions to try and scope out the high level requirements and each time we "peeled the onion" the board filled up till our brains hurt.

Minimum Learning Product

However the great thing about WordPress is that you can run and install it yourself just as easily as use their on-line offerings. I realised that the biggest knowledge gap that we had was WordPress itself both how to develop themes to how to use it to publish articles. We then made one of the best decisions in my opinion of the whole project and decided that we would use an Amazon AWS instance to self-host WordPress and create blogs.metro.co.uk. This would be a place that we would release real code to throughout most of the project which was able to be used as a prototype that didn't have the hangups of being the full site but would allow us to learn alongside the content team throughout the project. It took us two weeks to get our first release out and it was pretty damn basic, even with a large amount of on-line resources and a very talented team do not underestimate the complexities of learning a new system and programming language. We managed to get something out and get some really valuable early feedback from the business.

Planning For Success

It was also at this time that we started to think around how best to optimise the front end for the multitude of devices that were starting to grow into a sizeable part of our audience. Responsive design was starting to become a pattern for overcoming these problems and WordPress gave us the ability to focus more on the front end as the core CMS was already built for us. Again we had to learn a completely new way of approaching the front end so we did a phase of development on blogs where we could start to learn by building product. We now needed to move on to the initial planning phase. I think the key learning from this part of the project is to not plan too little and not plan too much.  There were some set milestones and we kept it very high level but the visibility that it gave us was very useful especially where external vendors timelines could impact our project. The biggest question was how long the development would take and to be honest we just chose a date so it would be delivered before the end of the calendar year that would give us 14 two week sprints. I think this approach is as good as any at this stage of the project as no matter how much thought and effort you will put into it you will be wildly wrong. All this time I had kept a single dedicated resource on the blogs project who was working on automating the deployment and tweaks based on feedback. The more time that your developers can be writing code during the project the better and automation is the right way to approach this.

100+ Item Detailed Backlogs Are Unmanageable

Now comes for what I perceived to be the biggest waste in the project. We spent a couple of weeks trying to map out all of the requirements in multiple day long meetings and then put everything in a massive backlog. We even did our best to provide story point estimates. However we knew so little about the how and WordPress provided so much functionality out of the box that it was just confusing. Having a backlog with over 100 items on it is unmanageable and as we were going through a steep learning curve requirements quickly became out of date but no one wanted to delete them. The struggle was the business wanting to know exactly what they would get in 10 months time. This fear of letting go of that knowledge is the biggest step in the transition to an agile mindset. By focusing on the high level problems you will solve rather than specific details you can start to bridge this gap. You can understand their fear in the context of multiple late previous development projects deployed by the previous team. Doing it all again I would only focus on details for the next 2-4 weeks with high levels goal for everything beyond that. Prioritise the bits to break down by amount of learning and risk to the project and get on with it.

Don't Forget Retrospectives

During this part of the process we decided to go with SCRUM as our project management methodology as its iterative approach fits well with traditional project plans. We tweaked our process at the end of almost every iteration until we got it to cause the least amount of pain. Our main learnings were around confusion born out of initially splitting stories into tasks for multiple people that had lack of a clear owner. We solved this by just having story tickets that one person would own at a time. We also struggled with the translation of business requirements through a Business Analyst (BA) to the backlog. The BA was an acting Product Owner and one that lacked the domain knowledge which made his a hard job. If your developers aren't talking to the business or a domain based Product Owner then they aren't learning and this lack of knowledge can create confusion. Confusion is a major waste of time in the early stages of a project. It also inhibits learning that is essential to gain momentum through the later stages of the project. This enables developers to make better decisions that can shorten feedback loops which leads to faster development speeds and less waste. This agility come from a common understanding of clear goals alongside validated learning. I now needed to hire some new people and build a team to get through the bulk of this project which I will cover later.

Crossing The Chasm

Gradually as we went through the project the business got more comfortable with reporting based on ability to hit the initially outlined goals and a clear view of where we were within the project plan. Planning for success helped us deliver this project on time and under budget.

Over the past few years error pages of the Internet have got a lot more interesting. Page Not Found know as HTTP 404 has been at the front of this movement. Rather than the more staid errors of old developers have started to inject their own special kind of humour into them. I was really interested In the psychology behind these pages, I then read "Your Logo Is Making Me Sick" which explains that when people are angry with your brand the last thing you want to do is have your logo part of that negative experience.

Every product will at some point encounter a critical error and/or force the user to wait. Don’t let those moments become synonymous with your brand.

The other thing to note is the state of the project that this was developed at. We were aiming for a very aggressive MVP release date and the whole team were really pushing hard for this. We knew we wouldn't get it all right straight away with over 300k articles to migrate from multiple previous CMS's and a completely new responsive front end. I had spent the day browsing 33 Animals Who Are Extremely Disappointed In You to prepare myself for possible miss of the release date as the project was on a knife edge. Then it struck me for all those people that wouldn't quite make it to where they wanted why not give them something funny to look at to leave them with a slightly confused but happy feeling. Image rights can be a bit of a nightmare but Metro has some great content and having had a chat with the content team the dancing polar bear was selected. The Sunday before the project as due to go live the team were in the office in a slightly silly mood and I happened to skate in that day to blow a few cobwebs out. GIMP came out and the dancing polar bear was born. He was later pimped out with a 404 chain just to complete the effect. I distinctly remember one of my team going surely this can't be the most important thing we have to work on right now. However when you have been banging your head against big problems for a week a break and something to take your mind off what you are doing is exactly what is needed. The day we went live we were monitoring 404 errors and suddenly saw our 404 page shoot to the top of our most read pages. We realised that it was being done on purpose to share on Twitter and from then on a legend was born. To get #1 on Buzzfeed's "The 28 Best Error Pages On The Internet" was just the icing on the cake. Thanks to @stephanfowler for the GIMP skills.

When I joined Metro they were using a custom CMS that used GWT for the admin screens which wrote to a CRX content repository and then used Cocoon to pull fragments of XML out and parse them into HTML for the  front end via ESI tags on Akamai's content delivery network (CDN). It was an incredibly complicated system that had started with big promises about the "future of semantic publishing" and was originally planned to be rolled out across Associated Newspapers as the default CMS. However the internal department which was in charge of development became Mail Online, the people who wrote their dissertation on it left the company and the project was left to myself and the team at Metro to stabilise and maintain. During this time it was taking on average three months to get a developer up to speed with the platform and our ability to move quickly was pretty much non existent. The implementation of CRX had also been setup by people who didn't really understand it's inner workings properly. Having spent most of a holiday I had on the phone trying to stabilise the system I came back with the knowledge that things had to change. At the same time my boss Jamie Walters had used WordPress.com to setup a blog that had a mobile, tablet and desktop version in less than 3o minutes.

CMS Selection Process

We then went into a large scale look at all of the publishing based CMS systems and quickly realised that spending a lot of money to tie yourself into a vendor who's main goal is to make money from you by making you upgrade every few years would not be the way forward. As part of that process we saw that Automattic the company behind WordPress.com had a VIP service that allowed you to utilise the might of their WordPress.com infrastructure  as a platform for publishing. This would allow the Metro development to focus on the user experience of news and be able to leverage all of the open source knowledge around the subject.

WordPress VIP's pricing structure was also very nice and simple with unlimited traffic, bandwidth and storage and a set charge additional charge depending on the support tier you require. They also helped out with our content migration and we ended up going with a high level of support throughout our build and then dropped it down once we had gone live. They also run six data centres in active-active mode so you don't need to worry about disaster recovery. Automatticians are a very dedicated bunch and are distributed throughout the world which really helps when working on large scale projects such as this. They also have the ability to change the WordPress platform to ensure that you can achieve your goals. One thing to consider is that they check every line of code before it goes up on their platform both before your code goes live and once it is up and running. WordPress is also constantly being updated by the large worldwide community it has behind it and a lot of the commodity items that a websites needs are taken care of for you. It had to go to the investment board twice but just before the second time CRX melted down and for five days straight we could only run it for 20 minutes out of every hour to update our Akamai cache and keep the site looking somewhat fresh. This couldn't have come at a better time in the decision making process and we got the sign off we needed.

Things worth considering when switching to WordPress VIP

We are now two months post our go live date and I can safely say that utilising the WordPress VIP platform has been a really positive experience. There are a few things that are worth bearing in mind however.

• WordPress.com does not support the www subdomain so we are now metro.co.uk rather than www.metro.co.uk.
• Initial Automattic code review can take 6-8 weeks to fully complete.
• Every time you make a change to your theme it needs to be approved by an Automattician (most completed within an hour and you can request faster in an emergency).
• WordPress.com does not support the passing of query strings around the application due to their caching setup.
• It is pretty hard to get your local and test builds similar to WordPress.com so get your code on their servers as soon as possible.
• The WordPress platform changes pretty much every day which has great benefits but you need to stay on top of updates.
• As most people who build WordPress sites do it for multiple clients it took us a long time to find someone to join on a permanent basis with the right level of skill.
• If you have any resource managing your servers currently then they will be less busy.
• WordPress VIP also offer self hosted help if you already have everything setup.

For how we approached the project from a project management point of view read some more here. Links:

• WordPress VIP Metro Migration
• WP Daily: UK’S 3RD LARGEST NEWSPAPER GOES WORDPRESS VIP
• Metro Responsive Website Launch now with added Swipe

First step was to update the DNS settings and change the IP of www to point to the IP address of my WordPress instance on Amazon EC2. Whilst waiting for this to propagate I needed to update all of the settings in WordPress to point to the new domain. I use the WordPress MU Domain Mapping plugin for this so navigated to my Network Admin > Settings > Domains and added a new entry to point to www and made it the primary for that domain. I then logged into MySQL workbench and ran the following.

update [dbname].[wp_site_id]_options
   set option_value = replace(option_value, '[oldsiteurl]','[newsiteurl]');

update [dbname].[wp_site_id]_posts
   set post_content = replace(post_content, '[oldsiteurl]','[newsiteurl]');

update [dbname].[wp_site_id]_posts
   set guid = replace(guid, '[oldsiteurl]','[newsiteurl]');

update [dbname].[wp_site_id]_postmeta
   set meta_value = replace(meta_value, '[oldsiteurl]','[newsiteurl]');

Waited around four hours for everything to propagate and Cafe On The Rye lives again in it's new responsive WordPress form.

UPDATE: Swipe was removed from metro.co.uk in October 2013 due to these reasons.

The Metro.co.uk site will become the first UK newspaper destination to allow mobile and tablet users to swipe between articles when it launches

Metro Ramps up Mobile Drive, Marketing Week

Linda Grant, the managing director at Metro, said the newspaper was the "original mobile product" when it launched targeting London Underground commuters in 1999.

Metro to launch new website, Android app and Kindle Fire edition, The Guardian

"We've taken app functionality and applied to browser product," - Jamie Walters, Product Development Director, Metro

Metro Announces Mobile First Strategy and Responsive Site, Journalism.co.uk

“We looked at Drupal and other platforms, but WordPress met our needs,” said Walters. “We no longer have to maintain back end systems.”

Metro Newspapers To Address “Smart Boredom” With Swiping, Techweek Europe

Free urban newspaper brand Metro revealed this week it is experimenting with new ad formats such as mobile payments and full-page interstitials between swipes on its new website as it looks to boost the utility of the ads it presents its readers.

Newspapers finally living up to digital first aspirations, Marketing Week

UPDATE 24.02.2013: The Metro responsive WordPress project was launched on time and under budget and here are more details about how we approached this from a project management point of view. This was also be the first national UK Newspaper with a responsive site and the first for the swipe interaction across all devices that support it.

Backup Wordpress to Dropbox

Install Dropbox

Download the latest stable Dropbox.

32bit

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86" | tar xzf -

64bit

cd ~ && wget -O - "https://www.dropbox.com/download?plat=lnx.x86_64" | tar xzf -

Lets start Dropbox

~/.dropbox-dist/dropboxd

You should get a message saying your account isn't syncing and to visit a URL. Copy the URL to a browser and login. Wait for the welcome message and then use Ctrl-C to exit. Now lets ensure that Dropbox will start when we restart our instance. Create a startup script.

sudo vim /etc/init.d/dropbox

If you are using a stock Amazon Centos OS then copy the below. If not check for a script at the bottom of this page.

# chkconfig: 345 85 15
# description: Startup script for dropbox daemon
#
# processname: dropboxd
# pidfile: /var/run/dropbox.pid
# config: /etc/sysconfig/dropbox
#

### BEGIN INIT INFO
# Provides: dropboxd
# Required-Start: $local_fs $network $syslog
# Required-Stop: $local_fs $syslog
# Should-Start: $syslog
# Should-Stop: $network $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start up the Dropbox file syncing daemon
# Description: Dropbox is a filesyncing sevice provided by dropbox.com
# This service starts up the dropbox daemon.
## END INIT INFO

# Source function library.
. /etc/rc.d/init.d/functions

# To configure, add line with DROPBOX_USERS="user1 user2" to /etc/sysconfig/dropbox
# Probably should use a dropbox group in /etc/groups instead.

[ -f /etc/sysconfig/dropbox ] && . /etc/sysconfig/dropbox
prog=dropboxd
lockfile=${LOCKFILE-/var/lock/subsys/$prog}
config=${CONFIG-/etc/sysconfig/dropbox}
RETVAL=0

start() {
   echo -n $"Starting $prog"
   if [ -z $DROPBOX_USERS ] ; then
      echo -n ": unconfigured: $config"
      echo_failure
      echo
      rm -f ${lockfile} ${pidfile}
      RETURN=6
      return $RETVAL
   fi
   for dbuser in $DROPBOX_USERS; do
      daemon --user $dbuser /bin/sh -c "~$dbuser/.dropbox-dist/dropboxd&"
   done
   RETVAL=$?
   echo
   [ $RETVAL = 0 ] && touch ${lockfile}
   return $RETVAL
}

status() {
   for dbuser in $DROPBOX_USERS; do
      dbpid=`pgrep -u $dbuser dropbox | grep -v grep`
      if [ -z $dbpid ] ; then
         echo "dropboxd for USER $dbuser: not running."
      else
         echo "dropboxd for USER $dbuser: running (pid $dbpid)"
      fi
      done
}

stop() {
   echo -n $"Stopping $prog"
   for dbuser in $DROPBOX_USERS; do
      killproc ~$dbuser/.dropbox-dist/dropbox
   done
   RETVAL=$?
   echo
   [ $RETVAL = 0 ] && rm -f ${lockfile} ${pidfile}
}

# See how we were called.
case "$1" in
   start)
      start
      ;;
   status)
      status
      ;;
   stop)
      stop
      ;;
   restart)
      stop
      start
      ;;
   *)
      echo $"Usage: $prog {start|status|stop|restart}"
      RETVAL=3

esac

exit $RETVAL

Now we need to add a file which contains the ec2-users that we want to run the service.

sudo vim /etc/sysconfig/dropbox

Add the following line to the file and save (:wq!)

DROPBOX_USERS="ec2-user"

Now we need to sort the file permissions.

sudo /bin/chmod 0755 /etc/init.d/dropbox
sudo /bin/chmod 0644 /etc/sysconfig/dropbox
sudo /bin/ls -l /etc/init.d/dropbox /etc/sysconfig/dropbox

Turn it on

sudo chkconfig dropbox on

Check that it worked, look for Dropbox in the list.

sudo chkconfig --list | egrep '3:on|5:on' | less

Start the service

sudo service dropbox start

Hopefully everything should now be working fine.

Move Dropbox Folder

I moved my Dropbox folder to my EBS volume so it would persist, then symbiotically linked it back to the home directory.

sudo service dropbox stop
sudo mv ~/Dropbox /vol/Dropbox
sudo ln -s /vol/Dropbox ~/
sudo service dropbox start

Now it is time to tidy up the directories where Dropbox sit and ensure that only the folders you want synced are turned on.

Turn On Dropbox Selective Sync

Go to your home directory

cd ~

We now need to download the Python Script which is used to manage Dropbox from the Command Line and give it the right permissions.

wget -O ~/dropbox.py "http://www.dropbox.com/download?dl=packages/dropbox.py"
chmod 755 ~/dropbox.py

Now lets check Dropboxes status

./dropbox.py status

Hopefully should now be syncing up, most likely you won't want everything to sync so you can use the 'exclude add' command to stop folders you don't need. Run the below and it will show your synced folders.

cd Dropbox
~/dropbox.py ls

You can stop each of them syncing by using the following command.

~/dropbox.py exclude add ~/Dropbox/Public
~/dropbox.py exclude add ~/Dropbox/'Camera Uploads'

If you run the below again you can see that they are dropping off.

~/dropbox.py ls

Also worth turning off LAN sync as you won't need it.

~/dropbox.py lansync n

Backup WordPress to Dropbox Including Database and Content

Create a folder for backups.

mkdir /vol/Dropbox/Backup

Now we want to write a quick script to copy the files from html/wp-content to Dropbox/Backup/wp-content and dump the MySQL. Create the below file, it should be synced to your other computers as soon as you save.

vim /vol/Dropbox/Backup/backup.sh

Here is the script, change the paths /vol/html/ to where you have your wp-content folder and /vol/Dropbox/Backup to where your Dropbox sits, plus update your MySQL password.

#!/bin/sh

echo "Backup Start"

echo "Syncing wp-content Files from html to Dropbox"

sudo rsync -vaz --delete --update /vol/html/wp-content/ /vol/Dropbox/Backup/wp-content/

echo "Backing up database"

mysqldump -uroot '-p[password]' --single-transaction --routines --triggers --all-databases > /vol/Dropbox/Backup/backup_db.sql

echo "Backup Finish"

Now lets run it to see if it works. (Update: previously I had used cp to copy all the files across every night. However I just noticed that it melts my server doing the update so have just updated to use rsync and just copy the files that have changed. Showing my Linux rookie credentials.)

sudo chmod u+x /vol/Dropbox/Backup/backup.sh
sudo /vol/Dropbox/Backup/backup.sh

Now we have our script sorted we need to create a cron job to run it once a day at 1am. You will need to stop cron.

sudo service crond stop

Now lets edit the file which runs our jobs.

vim /etc/crontab

Insert the below line, below this one # * * * * * user-name command to be executed:

0 1 * * * root ./vol/Dropbox/Backup/backup.sh

Save and start cron backup.

sudo service crond start

You can now rest easily that every night at 1am you can backup wordpress to dropbox including the entire wp-content folder plus a database dump. Perfect for the small time blog runner. Props to the following sites that were a big help in this process.

• Dropbox, Selective Sync
• Centos Dropbox Install
• Dropbox Wiki Text Based Linux Install

sudo find /vol/html/ -type d -exec chmod 777 {} \;
sudo find /vol/html/ -type f -exec chmod 777 {} \;

Once these have run you can run the update successfully, you then just put the files back to my what my previous post recommended.

sudo find /vol/html/ -type d -exec chmod 755 {} \;
sudo find /vol/html/ -type f -exec chmod 644 {} \;
sudo chmod -R 775 /vol/html/wp-content;

Happy updating, though if it doubt you can't beat the Hardening WordPress page on the Codex.

LeWeb London 2012 - Highlights and Key Takeaways

Day One

Jamie Oliver,  Television Personality & Kevin Systrom, Co-Founder & CEO, Instagram with Loic Le Meur, LeWeb  London 2012 Founder

I have to admit I am quite a fan of how Kevin Systrom has approached building Instagram and in the first big talk he was alongside Jamie Oliver.

Key Takeaways:

• Photo's bridge gaps that languages can't
• People tend to be much nicer when using images than when using words
• Instagram has democratised the creative process
• Photo's on Instagram tend to have around a 10 hour interest window
• Instagram now has 50 million+ users
• Instagram solved the problem of how people communicate visually using mobile
• Instagram API has allowed them to keep their focus on mobile and let other people fill out their ecosystem
• Instagram are working on the ability to allow you to go back in time to revisit older photos

“It’s like an empty stage with a drum kit and guitars–businesses need to hire passionate musicians to use them.”

Jamie Oliver on social media

Phil Libin, CEO, Evernote & Loic Le Meur, Founder, LeWeb London 2012

Key Takeaways:

• Evernote has 34 million users
• They make money from happy people and currently 4% of people pay
• The longer someone is a user the more likely they are to pay which rises to a maximum of 25%
• Average $US per user per platform
  • $1.06 - Android
  • $1.44 - Windows Phone 7
  • $1.79 - iPhone
  • $2.01 - Blackberry
  • $2.18 - iPad
• 50% of Evernote's development time is spent on Apple 50% on all other platforms
• Evernote Food makes $6.73 a user due to its niche user base
• The 30% they pay to Apple is a bargain when you include the marketing you get from being in the App Store
• More than happy for users to use in app purchases as it reduces the friction of their payment process

"Spend every single minute building the best product, if Google or anyone else manages to build a better product then they deserve to win and the world will be a better place for it."

Phil Libin on competition

Jason Goldberg, Founder & CEO & Bradford Shane Shellhammer, Co-Founder & Chief Creative Officer, Fab & Michael Arrington, General Partner, CrunchFund

Key Takeaways:

• Jason Goldberg lost $47 Million dollars in his previous start-up
• Focus on being the best at one thing
• When they pivoted from a Gay Social site to an eCommerce platform they turned their old site off for three months to ensure they could focus on getting the new one right
• Build the product first then the company will follow
• Experiment to get the product right then iterate to improve specifics
• 50% of their members come from social sharing
• Friction is the most common problem for people to pay for things online

“It’s all about shifting from impressions to connections”

Bonin Bough from Kraft on the future of online advertising

Martin Varsavsky, Founder & CEO, Fon

Amazing talk on how Martin achieves the focus and lifestyle that he needs to run a very successful company at scale.

"Delegation means that things will never be done the way you want but it is the only way to buy you time to do the things you love."

Martin Varsavsky

Day Two

Riccardo Zacconi, Co-Founder & CEO, King.com & Eric Eldon, Co-Editor, TechCrunch

Key Takeaways:

• King.com has 12 million daily users, 50 million monthly users but focus is on daily numbers
• They launch 80 games a year on their website for testing and the ones that get traction are then added to Facebook around 10%
• Monetise via advertising, virtual goods and VIPs
• Casual games are the most popular
• Cross platform capabilities really helps drive growth
• Core demographics female over 25 with core players female over 35
• Built a platform which enables them to wrap games with a social layer easily to help them scale

Shakil Khan, Head of Special Projects, Path &Loic Le Meur, Founder, LeWeb London 2012

Key Takeaway:

• Facebook is copying Path on mobile!

"Facebook is building the cities, Path is building the homes."

Shakil Khan

Sam Shank, Co-Founder & CEO, HotelTonight & MG Siegler, General Partner, CrunchFund

Key Takeaways:

• Same day booking system, 100% focus on mobile
• Some hotels still require a fax for booking
• 8 seconds to pay, 3 taps and a swipe, was just 3 taps but was too easy to book accidentally
• No legacy and a very singular focus on mobile and same day booking allows them to differentiate

"Lack of focus is the biggest killer for startups. I named my company HotelTonight to enforce this."

Sam Shank

Baratunde Thurston, Comedian

Key Takeaways:

• He is a funny dude!
• The Onion vote on best headlines for major stories out of a list compiled by staff

Dr. DJ Patil, Data Scientist in Residence, & Josh Elman, Principal, Greylock Partners

This was one of my favourite talks and they both did an amazing job of showing how big data works and can help shape decisions and the specific example from Twitter from 13 minutes in is priceless.

"Data science is the new black."

Tim O'Reilly

Key Takeaways:

• Use data to have a conversation
• Use data to find inflection points and then optimise towards

Data Scientific Method

• Start with a question
• Leverage all your current data
• Create features and run tests
• Analyse and draw insights
• Let data frame the next conversation

Conclusions

So there was a lot to take in at LeWeb London 2012 and congratulations to Blippar for winning the startup competition. Had an interesting meeting with Teleportd who came third about how we might be able to use their social photo aggregation tool at Metro.

LeWeb London 2012 - My Learnings

It wasn't until a couple of days later when I sat down to compile this post that I really began to distil how to apply startup thinking to a more established company. However can any large company these days really afford not to think like a startup? The world is changing around us at such a pace and modern technology aids rapid disruption of just about anything, so no one should get complacent. Most of the questions that the startup competition judges asked, were around what problem were these startups trying to solve. The more concrete the answer the easier it was to see how effective each of these businesses were. The other really useful part of having a clear problem to solve was the focus that it provided in asking the right questions around how the product was being developed. Focus was a reoccurring theme throughout from both entrepreneurs and seasoned executives. The CEO of HotelTonight even went as far as naming his current venture to ensure that they weren't distracted from providing the best experience when you are in need of a Hotel Tonight.   When Fab pivoted rather than trying to keep their existing community happy they turned the site off whilst redeveloping to ensure that they weren't distracted at all. I can't imagine that these decisions were easy or to some people seemed very logical but are very Lean in their approach. The clear goals and focus also make it much easier to use data in a way to shape decision making. By having a clear focus on growing users Twitter was able to work out the tipping points to turn first time users into regulars. This enhanced their focus on optimising those parts of the user journey that provided these metrics until they reached hyper growth. User experience was another key theme throughout LeWeb London 2012, with so much choice available to end users if your product doesn't delight them on a regular basis then they aren't likely to stick around. This can be hard from a scientific standpoint to measure and optimise but focusing on keeping people happy has allowed Evernote and many others to grow very sustainable profits. Path's lazer focus on this area has also allowed them to differentiate from Facebook on mobile to the point that they are now copying them. There is always someone bigger out there but most likely they will have a much larger set of people to keep happy and that can end up detracting from the end product. You have to be constantly looking forwards and focused on improving rather than looking at other people. Copy cat startups got a pretty bad rap from stage participants and especially from investors. By doing something different and disrupting existing businesses your potential rewards are so much greater than being just another clone. LeWeb London 2012 encapsulated an amazingly diverse set of ideas and thoughts into a rather enjoyable set of talks that didn't really speak to the "Faster than realtime" theme but not sure that was the point. London as a startup scene definitely isn't the Valley but that is probably why it is interesting to many.

Key Takeaways:

• Solve a problem
• Ask the right questions
• Focus on the above to at all costs
• User Experience is a key differentiator
• Use data to drive conversations and frame decisions
• Happy users stick around longer and spend more money

My Story to Come

This story begins three years ago when I first joined Metro to create a development team. Prior to this all development had been done by a central group resource, they took at least 6 months to get anything done and when it was finally delivered, it wasn’t what you asked for in the first place. The lack of agility was seriously harming Metro’s ability to make the most of product, commercial and editorial opportunities that were presented to them. It took the next 18 months to change not just the way that development was run, but also the way that Metro thought about development and the processes that guide it. Our first goal was to become Agile which is a word that is bandied about a lot but essentially means working in smaller iterations using small self-organising teams and regularly reviewing what and how we are working to continuously improve.

This path was not without its road bumps and the first major one was the development of the TV Product, this was an extremely complicated system which allowed people to write reviews of 15,000 TV shows. It took over six months of development and generated less than 100 reviews. As part of our Agile process we sat down at the end at realised we had to change even further. We needed to understand the drivers of ours users behaviour and ensure that we understood them before we over engineered a system which didn’t do what they needed. This is when we started to look at the Lean Methodology of manufacturing promoted by Toyota since the 1990’s and more recently by Eric Reis in his book The Lean Start-up. Being lean means that you limit the number of things that you are working on and look at each iteration more scientifically. For each iteration you have a customer based hypothesis that you build as quickly as possible before releasing and measuring its impact against your hypothesis, you then make your next decision based on the feedback.

Build, measure, learn, iterate became the development team’s mantra and ensured that we only built what was required and we learn't from every iteration. This ensured waste was kept to a minimum and constant feedback was able to help shape future developments. This enabled Metro to embrace the constant change that is present in our environment and use it as a competitive advantage. It didn’t take away the need for an end goal just ensured that the journey we took was as effective as it could be every step of the way. This quote I think sums it up nicely. “It is not the strongest of the species that survives, nor the most intelligent, it is the one that is the most adaptable to change”. It wasn’t just Metro who had used these principles Facebook have the words “Done is better than perfect” on their walls and both Facebook and Apple work in small teams specifically focused delighting their customers.

18 months ago Metro decided that they needed to take some brave decisions to diversify their business away from print revenues. With all of the learning that development team and Metro had been through over the previous 18 months and with each of the new start-up businesses having a technology component I decided it was time to break these agile and lean practices out of the development team and get them in use in the wider business. I realised that the best way to do this was to get involved with each of these new ideas and ensure that the both development and business resources used our template of build, measure, learn, iterate.

Boiling all of our new ideas down to a simple test and using data to shape our decision making process has been instrumental in changing the face of Metro to the one that you see today. We now always test our assumptions and if they are not correct, we fail early and quickly adapt. All data and experiments are shared through the use of highly visible dashboards. This has allowed Metro to collectively learn from each other and this level of transparency and openness has accelerated Metro’s decision making process and enabled it to stay far ahead of their competition. I would like to finish with a quote from Winston Churchill “Now this is not the end. It is not even the beginning of the end. But it is perhaps, the end of the beginning.”

Books that helped me shape this vision.

Radical Management by Steve Denning The Lean Startup by Eric Ries

Categories in my opinion are best used for the main structure of your site and can be thought about a bit like a menu. Each post should only be present in one category. Categories can also have  sub categories, this can be handy to split your content down into more specific areas when you have enough posts. A good example of this would be sport, for example for on a news based site you are quite like to have sport as a top level category, then under that individual sports such as football and cricket.

Tags in my opinion are best used to cut across categories and a post should have many tags but only one category. Tags have the advantage of being able to be combined to bring back more specific data when required. I think a good example of this would be a tag for "David Beckham", as he is both a footballer and a celebrity he could appear under either a sport or celebrity category and if you wanted to get more specific you could search for both "David Beckham" + "Victoria Beckham" if you are interested in their goings on or "David Beckham" + "LA Galaxy" if you wanted to know about his latest sport endeavours.

Ensuring that your taxonomy is clean and consistent can really help both people and robots navigate your site. Starting out in the right way with the right structure will lead to better results for everyone as changing later can be quite detrimental to any search juice that you have built up especially if you have the category name in your URL.

It is easy to get some experts in, go on a course or read how SCRUM should be done and implement it exactly as they have described. I have however found that the most important part of the Agile process is the retrospective and subsequent changes to your process. This self learning and feedback needs to happen not just within the development team but also with the business environment that they operate in.

A lot of processes assume certain resource requirements and business buy-in. However this is rarely the case when you start out. There are a lot of companies that say they do agile but aren't really agile more like Waterfall using SCRUM. Another major part of Agile is self organisation so someone managing the process too tightly stops this happening. Though at the beginning it is obviously good to start from a template and iterate from that.

It has taken me over 18 months to get both my team and the business in a place that they don't just get Agile but they are Agile in their approach. Quite a bit of this was born out of making mistakes along the way and the specific learning that we were able to distill from those. We can still improve but that is part of the appeal to me.

Speed of adaptability is essential in the modern business world to stay competitive. It is also immensely rewarding to see a team come together and improve over time. Two weeks ago we released early for the first time in our history. I am sure it won't be the last.

Support

Only one developer has any idea of how Node.js works and when there is a problem as there will be one when you are using cloud based hosting. This could be slightly mitigated by having a Wiki. However usually it is out of office hours during weekend support when you find out the Wiki is not quite 100% up to date.

Consistency / Learning Curve

Teams change and as we move forward we need to ensure that our code is written and approached in a consistent way. This will ensure that if a developer wants to get involved with different projects they can easily jump between them. It also ensures that if the person who wrote the code isn't around supporting it is much easier. Yet another barrier to entry will also only reduce the ability to quickly up-size should the project or team need extra resources.

Agility / Speed

We are an Agile team and working in short iterations delighting our customers is what we do best. By being consistent we become more agile, it aids self forming teams and enables continuous learning. A common approach make it easier for the knowledge to be passed around which amplifies its effect. Working on similar projects allows my team to take those learnings and get faster at doing what they do. The libraries and helpers that are developed can be used in as many places as possible to reduce code duplication and increase testability.

History

My team inherited a custom written CMS solution that we have been supporting for the past 18 months. It was written using Apache Cocoon and GWT as a front end with an Adobe CRX NoSQL solution as the data store. Due to none of these technologies being widely known development was up to ten times slower than it should have been because of the learning curve and lack of consistency. We also chose to use MooTools as a JavaScript framework for our front end and this has restricted the interoperability of our site and it doesn't get along with jQuery which is what the rest of the web uses.

Lack of a clear benefit

Node.js is a great solution if you have requirements for massive concurrency. However that requirement for our current problem as it will be cached behind Akamai. In this case there is not enough of a business reason to do something different.

As you can see I thought long and hard about this and I hope my conclusions make sense. I am really glad that we now have the knowledge of how Node.js works and when a project comes us that does have a clear benefit I will reconsider. However until then we are going to go with the obvious choice and rewrite this API in Spring MVC to be slightly less bleeding edge.

UPDATE Dec 2013: Looking back on the above post I still think it was the right decision at the time but the flip argument to all of the above is that it would have been great to learn some new technology and get the team enthused about this. Small distinct projects are a great way of trying out new bits of technology. Sure there is a risk associated if it doesn't work out. However if you aren't trying new things then you will never know when something better comes along. If I had to make the same decision again I would probably go the other.

Specifications

• Software: Windows Phone 7.5 Mango
• Processor: 1.5GHz
• Memory slot: No
• Display: 4.7in 480 x 800 pixels multitouch
• Connectivity: GSM/GPRS/EDGE/HSDPA, Wi-Fi 802.11b/g/n, Bluetooth, A-GPS
• Ports: Micro USB, 3.5mm headphones
• Camera: 8 megapixel, F2.2 lens, dual LED flash, and BSI sensor
• Video playback: 3gp, .3g2, .mp4, .m4v, .mbr, .wmv
• Audio playback: m4a, .m4b, .mp3, .wma
• Radio: No
• Battery: Li-ion 1600 mAh
• Size: 132x71x10mm
• Weight: 119g

sudo yum install php-gd

After a lot of searching I found this great answer on Stack Overflow which pointed me in the right direction. Have added this step to my previous post for setting up WordPress as well.

sudo su
chown -R apache:apache /vol/html

I then set permissions to what the hardening WordPress guide recommends for my html root as all my WordPress files are there as I am running MultiSite with multiple domains.

find /vol/html/ -type d -exec chmod 755 {} \;
find /vol/html/ -type f -exec chmod 644 {} \;

As apache doesn't have a login I feel this is worth the risk though there is probably a better way to do this. I then added ec2-user to the apache group and changed the permissions of the wp-content folder to have group write permission 775.

useradd -G apache ec2-user
sudo chmod -R 775 /vol/html/wp-content

This allows FileZilla or any other program logged in as ec2-user the ability to change files and folders in the wp-content folder only. If anyone has a better way of doing this I would like to know. I am only using SSH and SFTP to access the server with key files.

• Add site
• Host: Ec2 elastic IP
• Port: 22 (will sort by default when you select next option)
• Protocol: SFTP
• Logon Type: Normal
• User: ec2-user

Open Preferences

• SFTP
• Add Key File select your secure key [.pem] from where you saved it

You will need to make sure that the ec2-user has the correct permissions to do the actions you require, here are my thoughts on the best way to approach this.

• Create Amazon Web Services (EC2) Micro Instance
• Setup Apache and MySQL
• Install WordPress
• Ensure all data is persisted to Amazon Elastic Block Storage (EBS)
• Install WordPress Multisite
• Map multiple domains to Multisite

Amazon very kindly give you an initial free 12 months on one of their micro instances so thought this would be a good place to start. Here are a couple of things to note:

• EC2 image data does not persist if the image is terminated (either by user or an issue on Amazon)
• You can attach an EBS volume to create persistant storage and attach to an EC2 instance

Step 1: Launch an Amazon EC2 Micro Instance and attach EBS and Elastic IP

• After registration login to AWS Management Console and go to the EC2 dashboard
• Select the closest Amazon region to where most of your traffic will come from (top left)
• Select EC2 then Launch Image
• I used quick launch the selected Amazon Linux AMI 32bit
• Create new security key and download it to a memorable place
• Then select Elastic Block Storage - Volumes
• Create Volume and make sure it is in the same zone as your Ec2 server
• Attach EBS to your EC2 (note the name that it gives your storage)
• Select Elastic IP's and allocate new address then associate the address with the Ec2 instance (note IP)

Step 2: Login to your server via SSH and start setup

I am using a Mac so I just fired up the Terminal and the steps I took are below though if you are using Windows you can follow a tutorial for PuTTY here.

• Fire up Terminal and go to the directory with your .pem key inside
• Change the file permissions of the .pem key to 400 by running the below

chmod 400 [keyname].pem

• You can now login to your server with the following command

ssh -i [keyname].pem ec2-user@[elasticip]

Being a newbie to Linux I had to have a quick look around for the basics of navigation and here are a couple of very basic commands.

• To see the files in a directory listed in alphabetical order

ls -l

• to change directory to a path

cd /var/www/html/

• to remove a directory or file

rm -f -r [file-name]

• change user to root so you don't have to type sudo each time

sudo su

Step 3: Setup MySQL with its data persisting to ECB

Login then change your account to root, then run the server updates.

sudo su
yum upgrade

Install MySQL.

yum install mysql mysql-server

Create a filesystem on the attached EBS volume as it currently doesn't have one (I used XFS as the demo I was following did the same) . My EBS volume was called xvdf rather than sdf which the management consoles said so use what you noted down when setting this up. XFS isn't installed out of the box you need to install and register before you can create the filesystem.

yum install xfsprogs
modprobe xfs
mkfs.xfs /dev/xvdf

We now need to make sure that this volume is mounted every time we boot the machine by running the following. We then create and mount the drive in this case I have called it vol.

echo "/dev/xvdf /vol xfs noatime 0 0" | tee -a /etc/fstab
mkdir -m 000 /vol
mount /vol

We now want to create data directories on /vol so MySQL can use these for data storage as they sit on EBS they will be persisted should the instance have an issue.

mkdir /vol/etc /vol/lib /vol/log

We now need to move the required MySQL files and folders from their default locations to the new one that we have created on /vol.

mv /etc/my.cnf /vol/etc/
mv /var/lib/mysql /vol/lib/
mv /var/log/mysqld.log /vol/log/

Now that the files have move we can create symbolic links (shortcuts) from where the default files were installed to their new locations on /vol.

ln -s /vol/lib/mysql /var/lib/mysql
ln -s /vol/etc/my.cnf /etc/my.cnf
ln -s /vol/log/mysqld.log /var/log/mysqld.log

Hopefully everything should be in the right place and we can now start MySQL.

service mysqld start

If anything goes wrong then check the logs. I would also check the data directories that you created to make sure that you can navigate to them.

tail -f /var/log/mysqld.log

We now need to change the MySQL root password to be something nice and secure and then run the secure installation as this will be a production environment.

/usr/bin/mysqladmin -u root password '[newpassword]'
/usr/bin/mysql_secure_installation

Now we need to log into WordPress and create the required tables

mysql -p
mysql> CREATE DATABASE wpdb;
mysql> GRANT ALL PRIVILEGES ON wpdb.* TO wpuser@localhost
-> IDENTIFIED BY "another-new-password";
mysql> FLUSH PRIVILEGES;
mysql> exit

Step 4: Setup PHP, HTTP and download WordPress

Now it is time to get HTTP and PHP installed by running the following.

yum install httpd
yum install php php-mysql
yum install php-gd

We need to enable the Apache mod_rewrite module for WordPress to run correctly. I used the command line text editor vim and you can find the cheat sheet here. You need to change AllowOveride None to AllowOveride All inside the DocumentRoot Directory Directive, normally <Directory “/var/www/html”>

vim /etc/httpd/conf/httpd.conf

• vim Cheats
• i enables edit mode
• esc finishes editing
• :wq! writes and saves file

We now want to create the html folder on /vol so all of our web files will also be on an EBS volume. We can then do the same as we did for MySQL and remove the folder before creating a symlink from EBS to our EC2.

mkdir /vol/html
rm -f -r /var/www/html
ln -s /vol/html /var/www/

Now we have a web folder we can download WordPress

cd /vol/html
wget http://wordpress.org/latest.zip
unzip latest.zip

As I wanted to run a MultiSite Multiple Domain site I needed to have the WordPress files in the root of the web server so the following moves them from html/wordpress to /html and then deleted the /wordpress folder to keep things nice and clean. I also renamed wp-config-sample.php to wp-config.php for ease of editing.

cp -rpf ./wordpress/* .
rm -rf latest.zip
rm -rf wordpress
cp wp-config-sample.php wp-config.php

Step 5: Setting up WordPress

WordPress is a pretty damn easy thing to setup however you do need to edit quite a few files in this step so I thought it would be a good thing to connect via SFTP to allow easy manipulation. Here is a quick guide to how to connect to FileZilla, once you have done that go to in FileZilla.

/vol/html

At this stage you will be able to connect and view files within FileZilla but most likely unable to write or create them. This is due to the fact you created all of the files whilst using the root account and the ec2-user doesn't have any permissions to change them. You will need to change permissions of each of the files and folders, you need to use Terminal to all users to write and update them.

cd /vol/
chown ec2-user -R html
chmod 755 -R html

You will need to change this back at the end anyways but here is the WordPress documentation on file permissions to understand what is going on and the risks that you pose should you not.

You can now download wp-config.php and make the below changes

define('DB_NAME', 'wpdb'); define('DB_USER', 'wpuser'); define('DB_PASSWORD', 'new-password');

define('AUTH_KEY', 'xxx'); define('SECURE_AUTH_KEY', 'xxx'); define('LOGGED_IN_KEY', 'xxx'); define('NONCE_KEY', 'xxx') define('AUTH_SALT', 'xxx'); define('SECURE_AUTH_SALT', 'xxx'); define('LOGGED_IN_SALT', 'xxx'); define('NONCE_SALT', 'xxx');

$table_prefix = 'wp_';

WordPress provide a very nice helper for the Salts here.

Now we need to allow the apache user access to html so WordPress can update plugins and files and change our file and folder permissions to what WordPress recommends here.

cd /vol/
chown -R apache:apache html
find /vol/html/ -type d -exec chmod 755 {} \;
find /vol/html/ -type f -exec chmod 644 {} \;

Now lets fire up httpd.

service httpd start

It is also worth using the below to ensure that httpd and MySQL run on startup.

ntsysv

Finally you will also need to add a Custom TCP rule to your security groups using the AWS console for port 80 to allow traffic.

Now for the moment of truth, browse to your Elastic IP, should you have any issues you can look at the logs by using the following.

tail 100 /var/log/httpd/error_log

There is no way I could have achieved the above with out some similar great guides and here they are.

• How to Set up DB server on Amazon EC2 with data stored on EBS drive formatted with XFS
• Installing WordPress on Amazon EC2 / CentOS in 20 Minutes

There are a couple of great guides for Multisite and Multi Domain Mapping below which worked fine for me with WordPress 3.3.

• Multiple Domain Mapping
• Multisite

Further Reading

Since writing this post I have done a lot to my server to optimise the running of WordPress here is some additional reading:

• Speed up WordPress Apache
• Backup WordPress to Dropbox
• Reduce WordPress Hacker Attacks
• Speed up WordPress Apache with Batcache Multisite