21 product guidelines forged while growing Metro.co.uk 400%

October 13, 2014 | Comment

For the last two years I have been focused on the design, build and growth of Metro.co.uk utilising the WordPress VIP platform. Our approach consists of constant experimentation with both product and content which has returned a large set of data mixed with editorial feedback. This has been refined into a list of product guidelines to help us remain focused on growth. These are based on my experiences and our audience so yours may differ.

1: Good editorial content will deliver more growth than any product based approach

With a single well written/planned/timed story able to deliver millions of page views and course through the veins of social networks for weeks this should be the number one focus.

2: Good UX turns the dial more than any product hacks

The better the experience of product and content the more likely people are to visit your site, share your content and form habits around its consumption.

3: The closer to the main content area of the page the more related the content should be

Our data has shown that the closer to the article body or top of channel pages the better contextually related content perfoms. Once you are below these areas users are more open to a wider set of content to continue their journey.

4: Where content is placed on the page is almost as important as the content that is placed there

Our testing revealed content placement is almost as important as content selection (as long as it is relevant and recent). This is one of the reasons we have moved to an algorithmic approach for large areas of the site.

Nothing beats the value of an editorially selected contextual link within the article body
The area just after article delivers a lot of value as users have finished reading and can be easily tempted into something else
Sidebars aren’t shown on mobile and banner blindness often turns them off for desktop users so they are not an area we focus on

5: Fill dead space with content, people like to scroll it’s the natural behaviour of the web

Our newsfeed delivers over 10% of the page views of our site, this is pretty impressive considering it used to be blank space at the bottom of every article and channel page.

6: Don’t mess with the natural way that the web works

We tried and failed with this during our swipe phase. 5-7% of users delivered 20% of our page views but that didn’t increase their overall time on site. However it complicated everything we built hampering our ability to learn fast. It also didn’t quite fit into commercial or editorial strategies. This frustration/learning was what inspired the algorithm and scroll based newsfeed you now see.

7: Algorithms are great but need help from humans to perform at their best

Simple algorithms are a great way to optimise editorial workflows especially around content positioning. However these are only as good as the data behind them. Often you have to wait for this to be gathered before acting on it. Using editorial intuition is a great way to shortcut this process. Especially if you can make it run off existing priorities then process change isn’t required to participate.

8: Whatever Google/Facebook ask you to do just do it

They deliver so much of your traffic don’t question, just do what they recommend.

9: Feed the beast

Google and Facebook are always hungry for quality content. Gaining momentum requires constant feeding. They both have overall scores for domain as well as article urls so focusing on keeping this high means a better chance to gain and then maintain momentum.

10: Think of every page as a funnel, you loose users as they scroll but the lower they get the more open to their next engagement they become

The higher up the page something is placed the more people will see it. However the lower down the page someone is the more open they are to being tempted by some more content, advertising or interactions (e.g. poll vote, comments)

11: A mobile first approach is a great way to approach product prioritisation

Most of our traffic comes from mobile rather than desktop so it is logical to prioritise. This has formed a major part of our growth strategy.

12: Goals need to be concise, measurable and focus on why

The more people understand the goal and are able to affect it the more powerful it is. A goal that contains a why will always beat a goal that just contains a what.

13: Product specific performance should be broken down to actions per daily active users for comparison

This gives a much better overview of actual performance. Allows you to take out traffic fluctuations, just make sure you have enough data.

14: A week seems to be the minimum amount of data required to see if a feature has worked

Due to fluctuations in traffic and browsing habits. Also good to look at monthly and quarterly trends over longer periods as quite often they exhibit patterns that aren’t found at lower levels. It was asking questions around unexpected trends/data that helped teach me most around product growth.

15: Distribute weekly reports to show trends and give your stakeholders an overview of how the product is performing

Have these scheduled to your team and stakeholders via email. Also very useful if you break something when fixing something else. Great safety net to minimise impact and spot any unexpected growth.

16: Any new feature needs to be taken in context of how it fits in the editorial work flow. The closer it is to the existing process the more likely it will be adopted.

The best way to change a habit is build off an existing trigger. New features that leverage existing habits will get much higher adoption than building new habits/process.

17: Consider the users current journey and their emotional state in all features

Segmenting users based on mindset is a great way to understand data. e.g. Social browsers are likely on a multi site journey in a chromed browser on a mobile device. So they are only looking for a single story from your site so optimise for that. No point in worrying about pages/visit focus on getting more return visits via a social follow.

18: When coming from social users are often looking to enhance their social status

Our top share buttons get clicked on 4 times more than our bottom share buttons. Social proof around number of others already shared also promotes more sharing.

19: When coming from search users are usually in a topic based mindset

More likely to click on related, in article links and masthead channel links. Continue to deliver great content around a niche to form habits. Particularly useful around passion centres e.g. Premier League clubs.

20: It’s better to have 100 amazing tag pages that look and feel like a destination than 10,000 that feel like they were made for Google

Quality trumps quantity every time, Google knows if you users are clicking through.

21: People click on headlines 4x more than they click images

This is why A/B testing headlines is a great idea. It is the single piece of the editorial process that can have the biggest impact on growth. We also have SEO and socially optimised headlines to ensure we cater to both needs.

These are the principles that I have applied to the product development of metro.co.uk over the past two years. The key takeaway is that constant experimentation is a great way to unlock growth if your environment supports it. The hard part is achieving that without adding too much complexity. Complexity inhibits your ability to learn and learning is central to any successful product growth strategy. Building a set of guidelines has enabled us to move faster and helped foster our continued growth.

One for the future.

Micro interactions help drive habitual use

We don’t have a lot of data on this yet but there seems to be a correlation between micro interactions such as poll votes and habitual use. My theory is that by engaging different parts of the brain you become more memorable. These simple actions form the basis of new habits around content consumption. I think this is a major opportunity for future growth.

The thoughts and process behind Metro10

September 30, 2014 | Comment

This slideshow requires JavaScript.

Metro 10 was born out of a desire to experiment with native mobile news reading experiences that solved a different problem to our already fully responsive website. We had an algorithm that allowed customisation and decided to use this as our data source. The concept of restricting the volume of content would help us differentiate from our infinite web experience. We also wanted our users to engage with each article fully before moving on and not just get into a state of skimming which we had seen with infinite lists online.

Our goal was to minimise the time to launch so we decided a hybrid approach would help this. Utilise the native elements for caching and navigation but a web view for the article body. As our CMS stores all of the HTML and other markup in a single blob stripping this out was too much work for the first iteration of an idea that was unproven. Our initial ideas were based on simple mobile interaction patterns. Swipe to dismiss, interaction based personalisation and full bleed images as our tablet edition had experimented successfully with this. We decided on an Android first approach as we had Java skills in house and we also wanted to iterate quickly once it had launched. The turnaround in the Play store would allow us to do this.

I hadn’t written any code in four years and getting back in the saddle was a rewarding experience. Leaning a new language and platform at the same time took a little time but the tools and ecosystem are pretty mature. Initially I got out my little black Moleskin notebook and started sketching. Putting something down on paper helped me to start to visualise flow and basic function. I am no UX designer but I find sketching a really valuable first step in any design process.

Once we had a basic idea we jumped into a rapid prototyping phase. This involved a lot of copying and pasting from Stack Overflow and some working from home to get some focus to get the bare bones of an app together. It was buggy as hell but being able to show people on a phone and see there reaction was priceless. I then involved Matt our UX designer and he started to iterate on the design. We were lucky that the concept was simple and allowed very quick iterations. The design quickly morphed from a list that looked very web based to a full bleed image that felt more mobile first.

It was at this stage I got a few more of the team involved. When rapid prototyping it was good to be solo and really iterate quickly but once we were locked on an idea we needed stability and to start on a refactoring process to move towards a proper architecture. I believe in emergent design where you don’t over architect things up front but as you learn more of the systems and domain build what is most appropriate. This can save a lot of time as you only build what is necessary and refactor once you are sure of its value.

The next phase of development to our first alpha was frustrating at times but also very rewarding. We kept the requirements process very fluid as things would change on a daily basis. This was a challenge but having written user stories which were quickly out of date and fought with automation of testing we decided keeping things simple was the best approach. Guerrilla usability testing has been key to shaping the UX. As the initial idea was pretty out there putting it in real peoples hands in the real world was priceless. We quickly learnt that people liked the limited number of stories, enjoyed the full screen design but having to dismiss every story Snapchat style wasn’t something that they felt comfortable with. The paradox of choice was blinding them. With 10 possible stories to read and only one visible at a time and only one chance to read it they couldn’t decide whether it was interesting enough.

We decided to strip the user experience back to its most basic and only give people 10 stories at at time that they can scroll through left and right. With this approach people were quickly able to grasp the concept and understand the navigation. Calling it Metro10 ensured that we kept to our goal of just the 10 most popular stories right now. Throughout this process there have been a lot of tension between what constitutes our MVP. Due to a fixed timeframe we had to make some compromises around fluidity of design. The biggest was to leave all interactions via gestures rather than fluid animations. The level of complexity rose exponentially with these.

We were able to get validation of our ideas without them. Learning if the concept works is the most important and all of the underlying data, service and business layer need to work no matter what the front end looks like. Overall it has been a great experience in taking a concept from paper to reality in less than two months. There have been lots of zigs and zags along the way but that is what makes this job interesting.

MVP

This slideshow requires JavaScript.

Later iteration

This slideshow requires JavaScript.

Responsive Design and the Mobile First Mindset

August 12, 2014 | Comment

Image from: Brad Frost

Responsive design was a key enabler of the strategy that enabled Metro to grow to 34m monthly uniques. It put the end user first and ensured our content looked the best it could at the time of render. We worked from the mobile screen up to the desktop and this enabled us to make quick design decisions. Key principles like this are a cornerstone of rapid project delivery.

Changing the mindset of the business to make mobile a first class citizen was a big challenge. However once they had agreed on a mobile first paradigm we were able to tackle the hard questions up front and once this allowed the development team to focus on how best to approach the responsive elements.

Image resizing was a key part of keeping our file sizes low and having a fast perceptive render. We also dropped our custom font on mobile due to render speeds. An adopted an adaptive approach that only delivered the HTML necessary for the page to render and prioritised mobile for speed reasons.

Mobile first ensured we spent our time focussing on the main content that the user had come to read. We went for the simplest option of not including the sidebar of content on mobile as it would only add page weight. The volume of clicks in this area also justified it not being included. Users have a sidebar blindness and unless it is the main navigational feature will rarely get used as the brain turns it off to focus on reading the main content area.

We are currently going through a new redesign process which will give a new lick of paint to what we achieved two years ago but the principles remain the same.

The below is a presentation delivered at the Google HTML5 Roundtable about Metro’s journey into responsive design

Ten things we learnt building the Android App Metro10

March 25, 2014 | 7 comments

Metro10 is the first Android App that I have been involved in from concept to release in the Google Play store. It was designed, developed, tested and deployed in the first three months of 2014. We spent the entire process learning and here are details of 10 things that we learnt as part of the process.

1 – To begin the pen is mightier than the keyboard

Starting with a paper and a pen is a great way to iterate quickly through ideas. Sketches are one of the easiest forms of communication at this stage. There lofi form means you are usually more comfortable with changing them.

2 – Keep the concept simple

We went through many stages of adding complexity to Metro10 but we always came back to the simplest implementation. With the lack of space on a mobile screen and wide audience we intended to go for this was what kept redefining our approach.

3 – Rapid Prototyping is your friend

We managed to get a version on a phone within 3 days that had our initial concept on it. This helped us put it in front of a much wider set of people very early in the process. It didn’t run off feeds, ran out memory quickly and had random image sizes but was key to getting a wider set of people giving us feedback on our concept.

4 – Guerrilla usability testing is a great way to get early feedback

As we had a prototype early I just started going up to people and giving it to them without any instructions. Watching their frustrations and listening to the questions they asked provided amazing feedback to help shape our product.

5 – Image resource management on Android is not straight forward

We probably spent close to half the time on the app stopping out of memory errors due to cropping and displaying of images. By making sure you manually clear the copy of the image that is attached to the image view we managed to fix this.

6 – Do as much processing as you can on the server

We ended up using the WordPress image resizer to resize all of our images to the specific screen size before they got to the device. This helped speed up processing and ensured we only downloaded the image size we needed.

7 – 7″ tablets are actually treated as large phones

Android treats 7″ tablets as very large phones rather than small tablets. This can mean your app needs some adjustment to look good on the larger screen.

8 – Google Play Store Alpha and Beta Communities are the best use for Google+

Google allow you to set up a community on Google+ which will give pre-release access to members. This was a great way to get feedback and manage the roll out of the app. It really made this part of the process a pleasure.

9 – Unit testing and front end automation for repeatable testing

We used both JUnit tests and front end automation to quickly get feedback on every change. We moved away from the Android unit tests due to speed issues with the emulator. We managed to integrate those and the front end framework with our CI environment though the management of emulators again provided some interesting challenges.

10 – There is no substitute for using real devices in testing

Emulators provided a great way for us to get started quickly but using a mouse to control features proved tricky. We also saw different behaviour on emulators and real devices over the time period.

The best way to learn something is by doing it, by starting with a set of constraints and a focus on simplicity and I am very happy with the results we got with Metro10. We have now passed 500 downloads, are maintaining over 100 daily active users and still have over a four star rating.

How the Metro.co.uk Newsfeed Algorithm Works

March 10, 2014 | 10 comments

We have been working on automating large parts of the content ordering on Metro.co.uk since our responsive redesign in Dec ’12. This has grown from managing a few widgets across the site to controlling the majority of the homepage. The below describes the process we went through to achieve this.

The first step was to get data out of its separate repositories and in a place that allowed manipulation. We began to collect page views and social interactions from APIs provided by Twitter, Facebook and Omniture into a MySQL database. The basic initial equation we used to figure out what was popular was:

Popular = ((Tweets + Comments + Likes + Shares) * 25) + Views

We decided to give the social signals a significant multiplier as a share action gave a much stronger signal of how much the content was liked than a page view. Based on this thought process we started with a very high multiplier of around 50 but reduced this to 25 after looking at the results over a period of time. This was calculated every half hour after fresh data was collected. The main issue with this approach was the frequency of change within the top stories. So we changed ordering to be based on the rate of change between the previously calculated score. We called this trending and it gave a really interesting snapshot of what was popular on the site and this improved the frequency of change.

Trending = Popular Score (Now) – Popular Score (30 minutes ago)

When we redesigned the site this sat right below the top five stories on the homepage. When we check the stats we were surprised that the number of clicks were almost identical to the clicks on the top module that were editorially selected and also had very large clickable images. Clearly people were interested in what other everyone else was interested in. This data also proliferated across our sidebars and into our Tablet and Phone Editions via an API. It updated 24/7 and required no manual intervention so no matter what time you visited the site it displayed a mix of fresh popular content. Based on the numbers we decided to see if this could be extended to run more of the site.

Home Page Top Five Stories (Editorially Chosen)

Weekly Clicks On Top Five Stories on Homepage

Clicks On Top Five Stories on Homepage (Weekly)

Home Page Top Five Trending Stories (Algorithmically Chosen)

Clicks on Top Five Trending Stories on Homepage (Weekly)

One of Metro’s competitive advantages is that it runs a very lean digital operation. We want people to spend their time finding and writing great content rather than worrying about placement. A relatively small volume of our traffic goes to our channel pages (e.g. News) or looks at sidebars due to the volume of mobile traffic we pull in. So we decided to experiment further with this concept and get it to run the rest of the homepage. We had placed a timebased feed at the bottom of the homepage and had been very surprised with the number of interactions. Just being time based wouldn’t work for the homepage as we needed a measure of popularity, freshness and time.

I sat down last December and started prototyping approaches to take. We had to give new stories a chance to be featured before they had as much data. We also didn’t want one huge story to be at the top for too long. After much deliberation/time spent running around Hyde Park I decided I would use a coefficient based on story age to boost posts in the early stages and penalise them once they had been live long enough to have an unfair advantage.

(Views + (Social * 25)) * Hours Since Published Coefficient

Hours Since Publish	Coefficient
0	25
1	15
2	5
3	3
4	1
4-8	0.7
9-12	0.3
13-24	0.05
25-36	0.02

We had also been talking about how we could distribute content around the site better. Especially at the bottom of articles as this is where the majority of our traffic comes. This is when we placed the same timebased feed from the homepage underneath every article in what used to be empty space. We were amazed once again at the level of interaction this received and decided to optimise the data behind it.

Additional Pages of Data Pulled In Via Lazy Load for Timeline (Weekly)

The main idea was rather than restricting the stream to just articles from the specific sub-category (e.g. Sport/Football) you were in, we could use the highest level category and boost the subcategory you were in. Eg see all of Sport but have all Football stories closest to the top. This would give other very popular sports stories a place as well as aiding circulation around the site.

(Current Channel Boost + Views + (Social * 10)) * Hours Since Published Coefficient

I then added an option to allow you to pass in the channels that a user visited the most and using the same logic to prioritise their content near the top. Again not excluding content but rather ordering it in a way that was most appropriate for the person viewing. This is going to form the basis of our new Android App with it just displaying the top ten stories based on your browsing habits.

(Users Channels Boost + Views + (Social * 10)) * Hours Since Published Coefficient

The final twist was to use the data that we had collected on the backend to style the front-end. Our News Feed now changes its imagesize depending on if a story is trending or has been flagged by editorial. This give prominence to the popular stories as the algorithm pushes them further down the list and keeps the styling changing 24/7. As we use data from existing editorial placements it also means that there is no extra work to manage. We have worked on optimising the placement on this and we are now getting over 3 million News Feed lazy loads being requested weekly and over 500,000 extra page views. Considering this area of the site used to be blank this is a tidy return on investment. The latest enhancement is to grey out stories you have read to give an even bigger prominence to stories you have yet to consume.

Current Timeline Styling (Trending, Normal, Read, Editorial)

Clicks on Title in Timeline (Weekly)

Clicks on Image within Timeline (weekly)

The other large jumps in interactions were triggered by:

Changing the height on when additional content was pulled in
Increasing the size of images for trending stories
Putting comments behind a tab and making timeline the default view

The hardest part of this process was getting all the data in the right place from the APIs in the first place. Once this was present and we had a nice API around this the rest of this process was a iterative effort between editorial and development. There are a lot of moving parts but the benefits of having a site that reacts to users behaviours and works 24/7 is something that we feel is hugely worthwhile.

10 growth hacks that helped Metro.co.uk achieve 27 Million Monthly visitors

March 4, 2014 | 2 comments

Metro Monthly Unique Visitors (Jan’12 – Feb’14)

Over the past 12 months Metro has been on an amazing growth curve. Some of it is being in the right place at the right time for algorithm changes but a lot of it was planning and then execution of a growth (hacking) strategy.

Hack 1: Responsive Design

Monthly Unique Visitors from Mobile (Jan’12 – Feb’14)

We decided a responsive design would be the best way to capitalise on the explosive growth in mobile. A nine month redesign process culminated in Metro going responsive on 7th Dec ’12. We immediately saw growth from social referrals with Twitter’s almost doubling over night. The other benefit of responsive sites is that there is only one URL. As this is the key used to store ranking information in search and social algorithms you don’t want this split between multiple domains like m dot.

The key takeaway is that if you give people a great experience on all devices then they are much more likely to read, share and return to your content.

Twitter referral visits Nov '12 - Feb '13

Weekly Visits from Twitter (Oct’12 – Feb’13)

Hack 2: Focus all development efforts on growth

WordPress.com Stats for Metro (Dec’12 – Feb’14)

We also migrated to the hosted platform as a service provided by Automattic on vip.wordpress.com. This enabled all of our costs and resources to be focused on growth as they didn’t have to worry about caching, servers or anything that didn’t improve experience for our readers or editorial users. The amazing thing about the platform is that it is a flat fee. This means that although our traffic has grown 350% year on year our costs have not changed. The depth of their out of the box features plus ecosystem of plugins ensured that we did’t have to worry about commodity features such as SEO, site maps and editorial workflows as someone else had built and open sourced an approach.

Hack 3: Open up content creation to anyone

Metro Blogs Monthly Unique Users (Dec’12 – Feb’14)

The other great thing about the WordPress platform is that it enabled us to allow bloggers to contribute content directly into our core CMS. This started out as a feature for Metro employees but grew to encompass a much wider set of sources. Club Metro is now the most prominent and now contributes over 1M unique visitors a month. Having blogs content on the same domain placed amongst the rest of Metro’s content ensured we leveraged our existing algorithmic rankings. A single editorial workflow also helped keep the overheads low. The added bonus that most bloggers were already using WordPress and can write from anywhere helped us secure some top talent.

Club Metro Article

Hack 4: Facebook page as a major marketing channel

173,000 Facebook Page likes Feb ’13 to 562,000 Facebook Page Likes Feb ’14

Social referrals were another large contributor to growth. We focused on growing our Facebook page “Likes” as a content marketing strategy and this was very successful. We had always been careful to only send a small amount of our best posts out a day on our Facebook page. This had a solid base of users and growing this was a key goal for the last year. We employed many strategies including competitions with like gates and boosting posts. Competitions were effective in bringing in users but there were a lot of repeat entries. Varying the prize helped to minimise this and our email based CRM platform really helped to drive entries. The most cost effective way we found was boosting posts as friends of people who already liked Metro were shown a great piece of our content. They were then much more likely to then go on to like the page. As they were similar to people who already liked Metro they were very receptive and continued to engage with our content.

Metro Facebook Page

Hack 5: Made to share content

Weekly Social Visits to Metro (Jan’13 – Feb’14)

Ensuring that not just the content put out on the Facebook page was made to share also really helped grow our social referrals. When the content is written they set a success measure of 100 shares on an article and then set performance targets on achieving that on 25% of our content. This has allowed some interesting conversations around the areas of the site where that was less prevalent. More than anything it is a very easy test for all of the content creators to know what they should focus on. If it won’t hit that bar then find something else. A key growth hack we developed was the ability to show different headlines for social and search so we didn’t impact our search traffic.

SEO and Social Headlines on Made to Share Content

Hack 6: Made to share UX

Share and follow functionality on Metro.co.uk after the Made to Share focus.

The other side of social was increasing the number of people sharing from the site. The development team had a made to share focus where we introduced much larger, clearer social buttons and reduced the number of clicks it took to share. This with the addition of a sticky sharing bar that floats on desktop has seen a large increase in the number of direct shares from our site. This seems to have also affected the amount of people copying and pasting links from the site from the subtle reinforcement due constantly present share cues. It would also seem that Facebook take direct site shares as a strong signal in their algorithm as we have continued to see growth from social.

Hack 7: If something feels wrong don’t give up on it

Referral from Natural Search (Dec'12 - Feb'14)

Referral from Natural Search (Google Fixed, July ’12)

After the redesign only 20% of our stories indexed in Google News had our pictures next to them. We spent months experimenting on different options before we finally managed to ask Google the right question in July ’13 so they could fix it for us. It turned out that they were still using the Webmaster Tools account we had been using before the migration which pointed to our old domain that included www. Not only did this change help our referrals from Google News but it gave us a major kick in all search referrals. It would have been much easier to give up on this earlier but relentlessly focusing on this until we solved it really paid dividends.

Hack 8: Let technology automate repetition

Metro Development Releases to Production (Nov’12 – Feb’14)

Automation of all of our development and test processes allowed us to release 4.5 times on average every day (apart from Fridays) for the past 12 months. This enabled short feedback cycles and decision making to happen at a much faster pace. We have five different environments that are used for testing before we push code live. This kept errors to a minimum and kept the feedback flowing. This environment of automated front end tests and frameworks was a major investment but has continued to pay dividends.

Hack 9: Ensure proximity of key people who are focusing on a goal

Single Goal: 700,000 Average Daily Mobile Visitors in September 2013

A single goal of growth allowed us to work together cross functionally and a focus on data and numbers ensured that feedback was alway digestible. Content, social and tech sitting together and working together enabled the good ideas to come to the top quicker and equally the bad ones get ignored. Equally focusing on data helped take emotion out of decision making which enabled data to win arguments. This sped up innovation and focus. In most cases we have done less but done what we have done better to achieve growth.

Hack 10: Get out of the way

Voticle: Are you a true Brit?

Once people are working cross functionally together towards a goal then get out of the way and let them get on with it. In the past two months we have relaxed our process and now the content creators are working directly with the developers on new article formats to continue our growth. Out of this we have developed five new ways of displaying content from quizzes to lists and beyond.

Quizicle: How much of a Londoner are you.

Conclusion

None of the above would have been possible without the adoption of a lean mindset and the approach of build, measure, learn, iterate. It has been an amazing 12 months of growth at Metro and an great feeling to be part of a team that came up with and then executed a plan which delivered these results.

It was a pleasure to work with the below as well as many others on this journey.

Martin Ashplant @mashplant
Jack Dearlove @jackdearlove
Richard Moynihan @richjm
Gareth Noon @andfinally
Matteo Gildone @domsmasher
Matt Humphreys @mhumph
Hoque Ali @hoqueali
Paul Kevan @pkevan
Carlos Esteves @kalej
Dominic Rowell @domrowell
Jamie Walters @jamiewaltz

Music to write this code to

SQL script for easy 301 redirect WordPress blog htaccess

February 8, 2014 | Comment

I have just setup some 301 redirects in order to remove /development from the URL of my blog and wrote a handy piece of SQL to simplify 301 redirect WordPress blog htaccess. Once you have made the changes you can update your redirects /wp-admin/options-permalink.php. The reason I am redirecting to the guid which is the id based form is in case I change the slug in the future.

SELECT CONCAT('Redirect 301 ','/slugtoreplace/', post_name, ' ', guid)
  FROM wpdb.wp_posts 
 WHERE post_type = 'post'
   AND post_status = 'publish';

You can then post the output from that script into your .htaccess file and everything will redirect properly.

Music to write this code to

Claude VonStroke’s Essential Mix minus all of the Pete Tong guff in all of its bass heavy glory always gets me at the command line.

SQL script to restore WordPress backup localhost

February 1, 2014 | Comment

I needed a simple SQL script to help me restore WordPress backup localhost to ensure that I didn’t have to go through the pain every time that I wanted to bring my data to my local server. This worked really well and all you need to do is search and replace on local.blog to the name of your localhost.

-- update site
UPDATE wpdb.wp_site
   SET domain = 'local.blog'
 WHERE id = 1;

SELECT * 
  FROM wpdb.wp_site;

-- update blog details
UPDATE wpdb.wp_blogs
   SET domain = 'local.blog'
 WHERE blog_id = 1;

SELECT * 
  FROM wpdb.wp_blogs;

-- update site meta
UPDATE wpdb.wp_sitemeta
   SET meta_value = 'http://local.blog'
 WHERE meta_key = 'siteurl';

UPDATE wpdb.wp_sitemeta
   SET meta_value = '127.0.0.1'
 WHERE meta_key = 'dm_ipaddress';

SELECT * 
  FROM wpdb.wp_sitemeta
 WHERE meta_key in ('siteurl','dm_ipaddress');

-- update options
UPDATE wpdb.wp_options
   SET option_value = 'http://local.blog'
 WHERE option_name in ('siteurl','home');

SELECT * 
  FROM wpdb.wp_options
 WHERE option_name in ('siteurl','home');

-- update posts
UPDATE wpdb.wp_posts
   SET guid = REPLACE(guid,'blog.david-jensen.com','local.blog');

I have gone through each of the tables above that need updating, you could probably get away without updating all of these but I figure that as it is a script you might has well do it properly. I have also included the select statements afterwards to ensure that you can see the value has changed.

Music to write this code to

Erol Alkan classic is something super chilled to sip your tea to for the calm before the development storm once you have your local restored.

Communication and trust are the foundation of high performing teams

January 27, 2014 | Comment

High performing teams can achieve unbelievable performance multipliers over sets of like minded individuals. I have spent the last few years working on building high performing teams and from this experience I have found the foundation of high performance is communication and trust.

It takes time

It is hard to get communication right within a team, even when they are sitting next to each other. Often it can take months rather than weeks to build up the right cadence of communication and understanding that are required to build trust. This process is even harder when you are also in the midst of creating a new company as it is only one of the many things that require focus. Luckily there are templates that help you hack these processes to make them happen quicker.

Agile mindset

Working with an agile mindset encourages regular feedback and can provide templates for the flow of information and cadence of communication. It also gives you the regular opportunity to tweak your process to ensure that it fits your environment in the form of retrospectives. Proximity is another clear hack and the closer that people sit the easiest it is to get them to communicate.

Standups

I have found that daily stand ups are a great way to hack communication within teams. I do daily stand ups with development teams and at least weekly stand ups with the business and stakeholders. These are best done around the agile wall and include new features as well as an overview of roadmaps. Regular feedback in the form of small regular conversations beats larger infrequent conversations as they allow for regular course correction. This helps prevent waste that comes from working on things that are not the most important for the business.

Consistency of language

The other benefit of regular communication is that both sides have the ability to learn the others language. Having a commonly understood language takes time and reduces the waste caused by miscommunication. Trust is much easier to lose than it is to build and in the majority of cases it is a miscommunication that leads to erosion of trust.

Find common ground

When people are under pressure to deliver it can be easy to blame something you don’t understand. There are many different cultures that make up an organisation and how you build a software system isn’t the same as how you create budgets. In order for cultures to understand each other you need to establish regular communication over common ground.

Have a clear vision

Common ground should be a clear vision based around a set of hard but achievable goals that everyone can affect. Clear goals enable decision making to be decentralised which helps to avoid one person blocking progress. Maintaining flow is essential to the creation of momentum which forms the baseline of high performance. Making your vision and current goals very visible should allow your team to see how the work that they are doing is affecting the key business metrics.

Conclusion

When teams are communicating well within a clear framework of success it allows everyone to focus on delivering value. The ability to influence business goals with regular feedback creates an ideal environment for high productivity. Communication is an often over looked discipline when forming teams and especially companies. It is the foundation of common understanding and this builds trust that will allow you take your performance from good to great.

Music to write this code to

Bonobo stepped up to the essential mix and the resulting two hours are outstanding.

How to install Varnish Cache on AWS Centos to speed up WordPress Apache

January 12, 2014 | 9 comments

I have decided to give my server the final boost in my quest for ultimate cachability and install Varnish cache. It was a toss up between doing this and installing NGINX but I have a reasonably large set of rules in my .htaccess to prevent hackers and 301s so I figured the migration would be a bit more than the few hours I had to spare.

First install varnish.

sudo yum install varnish

Make sure that it comes back on startup.

chkconfig varnish on

Now edit the following file.

sudo vim /etc/sysconfig/varnish

The below is what I ended up with in my file thanks to this. The main difference from the original is the change of the VARNISH_STORAGE_TYPE to malloc which means that it is stored in memory rather than disk. I chose 128MB as my blog is pretty small and I wanted to have a bit of room to spare.

# Varnish Users
VARNISH_RUN_USER=varnish
VARNISH_RUN_GROUP=varnish

# Should we start varnishd at boot?  Set to "yes" to enable.
START=yes

# Maximum number of open files (for ulimit -n)
# NFILES=131072

# Locked shared memory (for ulimit -l)
# Default log size is 82MB + header
MEMLOCK=82000

# # Maximum number of threads (for ulimit -u)
NPROCS="unlimited"

# # Maximum size of corefile (for ulimit -c). Default in Fedora is 0
DAEMON_COREFILE_LIMIT="unlimited"
RELOAD_VCL=1

# Should probably change this
VARNISH_VCL_CONF=/etc/varnish/default.vcl

# Not setting VARNISH_LISTEN_ADDRESS makes Varnish listen on all IPs on this box
# (Both IPv4 and IPv6 if available). Set manually to override this.
# VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=80

# Telnet admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082

# Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret

# The minimum number of worker threads to start
VARNISH_MIN_THREADS=50

 # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=5000

# Idle timeout for worker threads
VARNISH_THREAD_TIMEOUT=120

# Best option is malloc if you can. malloc will make use of swap space smartly if
# you have it and need it.
VARNISH_STORAGE_TYPE=malloc

# Cache file size: in bytes, optionally using k / M / G / T suffix,
# or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=128M

VARNISH_STORAGE="${VARNISH_STORAGE_TYPE},${VARNISH_STORAGE_SIZE}"

# Default TTL used when the backend does not specify one
VARNISH_TTL=60

# DAEMON_OPTS is used by the init script.  If you add or remove options, make
# sure you update this section, too.
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
             -f ${VARNISH_VCL_CONF} \
             -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
             -t ${VARNISH_TTL} \
             -w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
             -u ${VARNISH_RUN_USER} -g ${VARNISH_RUN_GROUP} \
             -S ${VARNISH_SECRET_FILE} \
             -s ${VARNISH_STORAGE}"

Then you need to edit your /etc/varnish/default.vcl the below is what I ended up with which allows you to see the WordPress Admin Bar but caches most other bits. It also takes invalidation requests from the Varnish WordPress Plugin which is the easiest way to keep your content fresh.

backend default {
  .host = "127.0.0.1";
  .port = "8080";
}

acl purge {
  "localhost";
}

sub vcl_recv {
  if (req.request == "BAN") {
    if(!client.ip ~ purge) {
      error 405 "Not allowed.";
    }
    ban("req.url ~ "+req.url+" && req.http.host == "+req.http.host);
    error 200 "Banned.";
  }

  if (req.request != "GET" &&
      req.request != "HEAD" &&
      req.request != "PUT" &&
      req.request != "POST" &&
      req.request != "TRACE" &&
      req.request != "OPTIONS" &&
      req.request != "DELETE") {
    return (pipe);
  }

  if (req.request != "GET" && req.request != "HEAD") {
    return (pass);
  }

  # don't cache authenticated sessions
  if (req.http.Cookie && req.http.Cookie ~ "(wordpress_|PHPSESSID)") {
      return(pass);
  }

  # don't cache ajax requests
  if(req.http.X-Requested-With == "XMLHttpRequest" || req.url ~ "nocache" || req.url ~ "(control.php|wp-comments-post.php|wp-login.php|bb-login.php|bb-reset-password.php|register.php)") {
      return (pass);
   }

  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (pass);
  }

  remove req.http.cookie;
  return (lookup);
}

sub vcl_fetch {
  if (beresp.status => 400) {
    set beresp.ttl = 0m;
    return(hit_for_pass);
  }

  if (req.url ~ "wp-(login|admin)" || req.url ~ "preview=true") {
    return (hit_for_pass);
  }

  set beresp.ttl = 24h;
  return (deliver);
}

You then need to update your Apache to listen to port 8080 as Varnish will be listening on port 80.

sudo vim /etc/httpd/conf/httpd.conf

Update Listen 80 to Listen 8080 and then restart Apache and Varnish and then you can run varnishstat to see the details of how your cache is performing.

sudo service httpd restart
sudo service varnish restart
varnishstat

I then installed the Varnish WordPress Plugin and then went to Dashboard -> Settings -> WP-Varnish and updated the following fields.

Varnish Administration IP Address
Varnish Administration Port
Varnish Administration Secret

The secret is a GUID which is stored in the below.

sudo vim /etc/varnish/secret

These are some really useful Varnish commands to see how your cache is performing or check out the docs for full details.

varnishtop: like top but for varnish.
varnishhist: histogram of request time
varnishstat: cache hits, resource consumption and many other data points
varnishlog: log of all requests made to the cache

You should then have a super charged server where most of the files are coming from memory rather than disk. The below sites were helpful along this journey.

Install Varnish HTTP Accelerator with WordPress
Speed up WordPress with Apache and Varnish
WordPress Varnish Cache Config / VCL

Music to write this code to

Makoto the drum and bass maestro does some house and techno which always gets me going at the command line.