How to speed up WordPress Apache using Batcache Multisite and Memcache on CentOS

Batcache
I run a small WordPress network on an Amazon Micro Instance and this means that I need to stay on top of caching. Having had a look around at a lot of different options I decided that I would use Batcache and Memcache to provide me with in memory caching should one of my posts get popular. Batcache multisite capabilities means that I don’t have to stress about individual caching setups per site. Also Batcache is what WordPress.com uses to handle caching and you can utilise Memcache in a distributed way should you need to scale out from a single server.

Another reason I like it is that it is very non-invasive caching technique, I know that WP-Super-Cache and the way that it generates static files is probably a better fit for my blog as the load isn’t usually very concurrent. However that sounded quite straight forward and I like a command line challenge plus I have a few side projects that this might come in handy for. You also need to get this setup if you are ever deploying anything to WordPress.com which we utilise for our hosting at Metro.co.uk.

As I am running a network, I don’t want other users to have to worry about caching and its quirks. The nice thing about Batcache is that it doesn’t cache anything when you have cookies from the domain so anyone who is writing can see their changes immediately. I think this and its simplicity is probably the key advantage over some other methods of caching.

The other thing is that if you have root access to your server then it is reasonably straight forward and I thought that I would share how I did this here in case anyone else is interested.

First install all of the libraries that you need to run memcache and access it via PHP, it took me a few times to get this right so there might be a few libraries which are extra below.

sudo yum install memcached
sudo yum install php-pecl-memcache
sudo yum install zlib-devel
sudo yum install php-pear
sudo yum install php-devel
sudo yum install libmemcached-devel

This is the compiler I used for building

sudo yum install gcc

This will build the libraries that PHP needs to be able to access Memcache

sudo pecl install memcached

Now make sure that memcached will start on startup

sudo chkconfig memcached on

I also enabled memcache session handling but haven’t turned it on.

Enable memcache session handler support? [yes] : yes

I didn’t do as it suggested below as every time I did I got errors saying that it had already been registered.

You should add "extension=memcached.so" to php.ini

I had a look at the config and tried to lock it down to localhost only but it kept on throwing errors when I did so I left it, as all of my ports are locked down via the Amazon Firewall this shouldn’t be an issue.

sudo vim /etc/sysconfig/memcached

Now made sure that it would come back on startup.

sudo chkconfig memcached on

Now I followed these instructions for Batcache installation once I had downloaded it.

Upload advanced-cache.php to the /wp-content/ directory

Add this line the top of wp-config.php to activate Batcache:

define('WP_CACHE', true);

Tweak the options near the top of advanced-cache.php, I set mine to cache for 60 minutes after being hit twice in 10 minutes.

var $max_age =  3600; // Expire batcache items aged this many seconds (zero to disable batcache)
var $seconds =    600; // ...in this many seconds (zero to ignore this and use batcache immediately)

Optional Upload batcache.php to the /wp-content/plugins/ directory. As I ran Multisite I put this in the /wp-content/mu-plugins so it would be there for all of the sites in my network.

We have already installed the PECL memcached extension so you just need to add Ryan’s Memcached backend 2.0 to wp-content.

Test by reloading a page in your browser several times and then viewing the source once you have cleared any cookies from the domain you are hitting. Just above the closing tag in the head you should see some Batcache stats.

generated 524 seconds ago
generated in 0.374 seconds
served from batcache in 0.002 seconds
expires in 3076 seconds

Now feel free to tweak the settings around cache time to a period where you will reduce the load. Remember this is full page caching and you shouldn’t see it if you are logged in.

SEO Impact

The best part of installing Batcache has been the impact on SEO, below are my crawl stats from Google and you can see that the average time spent downloading a page flatlines around the beginning of December after installing Batcache. The really interesting thing is that the pages crawled per day went up at the same time. I think this is because Google now thinks that that server can handle a much greater load so sends it its bots on a more regular basis. Can’t say that this has had a massive impact on my SEO entries but it is a really good sign of the impact that you can have with a proper caching strategy.

Google Crawl Stats

Super fast and simple caching, if its good enough for WordPress.com its good enough for me. The below articles helped me along this journey.

Music to write this code to

Nothing beats repetitive beats to get down on the command line and Sasha playing at DC-10 for Circo Loco knocks that out of the park.

Heidi Rozen: It’s the business model, stupid

Cogs
Heidi Rozen gave a great talk at the DMGT Technology Summer School simply titled “Failure”. A Silicon Valley Alumni she has started companies, worked for Apple under Steve Jobs and now is involved in Venture Capital so she had her fair share of stories. I was really impressed with her perspective on development and seemed to have a very Lean outlook after all of her experiences. I collated the points of her talk below and most of it resonated very well with how I am currently thinking.

  • It’s the business model, stupid – first place to start when a company isn’t working
  • Most technical decisions are actually business decision
  • It’s not just about money, it’s also opportunity cost
  • Support the lunatic fringe
  • Understand the assumptions so you can quickly adjust when things don’t work out and they don’t
  • Innovate around what is core for you and outsource the rest
  • You are building for the platform of tomorrow, not of today
  • You must understand your customers ecosystem and fit in effortlessly
  • If you can’t win then recognise early and deal with it
  • Think backwards from your customers
  • Narrow the customer set to provide 100% solution
  • Understand the ecosystem and the business model
  • Build 95% solution and pay 5% is Google solution
  • 60% of VC’s don’t return on their capital

Wendy Lea from Get Satisfaction on Customer Driven Development

Get Satisfaction

Wendy Lea the CEO of Get Satisfaction gave a presentation at the DMGT Technology Summer School on “The voice of the customer”. There were some great points on how they have infused passion for their customers into a customer driven development process. As their business is based completely on capturing customer feedback it was great to see how they have infused their company mission throughout everything that they do. I have collated the main points I took from the talk below.

  • “Listen -> Engage -> Iterate” is their mantra for development
  • “Advocate -> Discover ->  Buy” is a common loop for new customers
  • You need to create and curate a community for product development (Facebook, Twitter handle)
  • The Cluetrain Manifesto – “The connectedness of the Web is transforming what’s inside and outside your business — your market and your employees.”
  • Create systems of engagement mashing in your external and external systems
  • Conversations created by real people, in their natural language are very powerful and discoverable
  • Bidirectional conversation can help that flow
  • Join the conversation:
    • Embrace customer conversation
    • Open these conversations
    • Collaborate
    • Apply actionable insights

Embracing the above thoughts into a customer driven development process is something that really got me thinking on how we might be able to apply this at Metro. Apart from the old letters page newspapers traditionally are more of a one way dialogue. I have been thinking of ways that we could apply some of the above and simple feedback forms are something we are going to begin trialling soon as we make further changes to Metro.co.uk.

Strategic Perspectives from Bill Raduchel

DMGT Technology Summer School

I had the pleasure of spending some time at the DMGT Technology Summer School with Bill Raduchel. He had a long and interesting career before joining DMGT as a technology advisor to the board. I managed to capture some of his advice below and a couple of quotes which resonated well with me.

“Heroism is an act of stupidity that you happen to survive.” Bill Raduchel

  • Open source is the only way to get the best developers as they can learn portable skills
  • Proprietary systems are struggling
  • John Chambers CEO Cisco: We compete against market transitions not companies
  • You must get people loosely coupled but tightly aligned
  • You must have a vision and be able to sell it, you must communicate it in a way that all your employees can understand
  • Asking the right questions is never interference
  • You must set the boundaries and then get out of the way
  • Architecture is to empower your team, not constrain it
  • Commitment before success is the mentality that separates people in a startup from people in a corporation

“Mindset is a value that has become a habit.” – Bill Raduchel

How Metro built a metrics driven product development process

Target
I have spent the last three years at Metro constantly tweaking our development process to fit our environment and teams maturity. At the same time the product development process had remained reasonably unchanged. Someone comes up with an idea we all have a debate about how good/bad we think it will be and someones opinion wins out the day. However in the last nine months we have also begun to iterate on this part of our process to make it more of a metrics driven product development process. It has been a slower process than I had anticipated and thought I would share some of the learning around that here.

I think that the lack of established templates to follow such as SCRUM has made this a harder process as there isn’t a template to start from. I also think that in order to get it right the metrics need to be driven from the business, so you are dependent on them being engaged. The toolset that helps you move to this type of approach is also lacking and in the end we had to build/hack our own.

Our journey began by the business setting one key goal backed by a clear metric that the whole business could understand. This was easier said than done and were definitely parts of the business that feel left out by this singular approach. However just having one key metric in the beginning is very important as it aids focus. Metro’s was to have an average of 700,000 daily mobile visitors in September 2013.

The next phase was to have a look at all the data that makes up the key metric and to break it down further to show each area of the business what they could effect. Being as open and honest about where the numbers have come from at this stage is essential. There will definitely be questions about how the decision was made especially if it is a large goal and being open at this stage is the best way to combat that (even if it it is a vanity/stretch target).

As the largest part of the goal was going to come from metro.co.uk and that partly my teams responsibility we started by looking at our web analytics (Omniture) to see what data we were already collecting. We also looked to the content and search teams to see what metrics they were tracking as only by working together would we be successful. Common goals should bring people together to talk about frustrations/improvements, this alignment should provide major benefits. However it still surprises me in large businesses how little people actually talk.

I think part of this problem is the fact that so many parts of the business have their own language. Development as much as anyone contributes to this as it takes quite a lot of time to be able to converse with most developers. Marketing, Sales and Content all also have their own ways of working and language which can be just as daunting for developers. Getting the translation right between these different groups so they can talk enough of the same language to communicate effectively is a major business benefit. I think this is one of the reasons that startups are able to iterate faster as they have to talk the same language as you have a limited set of people doing a lot of of different things but all speaking the same language.

Having had and looked at all the existing metrics we decided that there wasn’t actually enough specific product based feedback mechanisms to measure the success of our work. So to understand the product more we had to build our own measurement/tracking framework on top of Omniture. This took some work to get right and then took even longer to get the tracking across our entire site. However once we were able to see the results of each of our improvements in both click perspective and overall impact on the wider numbers a really important feedback mechanism was created.

Measuring the impact was only the first part however we needed to get the data into a format that everyone could see and understand. This was to be a common theme throughout this process you go from having no data to having too much data that doesn’t make a whole lot of sense. Having dedicated resources and time to create dashboards that compiled this data into meaning was the next step on this journey. Then we created regular meetings that talked around this data and tried to put meaning around it. Product Performance was born and we scheduled it in for every two weeks. We get all of the product owners and stakeholders that are involved in product development into a room and talk through performance of the previous two weeks. We also do a product based standup twice a week where all stakeholders stand in front of the Kanban board and discuss what is being done and about to drop into the process. Lots of small regular conversations definitely provide more value and leverage for change than large infrequent meetings.

We also got large televisions that monitored real time traffic and put them up in visible locations so everyone could see what was going on and feel the excitement when a large story took off. Real time data is the next step in this process and the faster you can see the data the faster you can react to it. The next step after having everything measured was to begin A/B testing. This step again proved to be a much harder than we initially thought. As our site is news based and constantly changing we found we couldn’t really use a Javascript based framework like Optimisely as things changes too much. Plus the cost was very prohibitive for a site of our size. So we ended up building a simple framework that extended our measurement in Omniture.

The other key learning was as we release multiple changes everyday what do you test and what do you just measure. We got to a place that if you were working on an incremental improvement then A/B test otherwise just ensure that everything is measured. We were surprised as well that sometimes the A/B results were 50:50 but when we made then change anyway it had a longer term affect on user behaviour that was positive and not captured during the testing.

We also had some issues with our Google News Sitemap and indexation. This was out most experimented feature over months. We ended up creating a dedicated swim lane on our Kansan board just for this. We only ran one test at a time and then iterated to the next one. I think for harder more complex options running just one test at a time really helped focus in the results of each before moving on.

Running too many tests at the same time proved to be confusing and I ended up making one of the team the chief growth hacker and he became the go to person for what and how to run tests. Now that we are in a place with a nice testing framework and approach it was time to see if it would work towards a more major site redesign. We needed to break this big design down into testable chunks that would prove our hypothesis that people would interact more by scrolling than swiping. Creating stories that were just for learning felt like a natural evolution from a lean mindset.

Conclusion

It has been a really interesting nine months and we have managed to hit a goal that seemed completely unachievable only 12 months ago. I think we managed to create a measurement mindset where everyone asks what we are measuring before starting development. This added to the power of everyone pointing in the same direction, having a process that allows constant improvement and promotes communication has produced some amazing results. You should give it a go.

  • Start with one key metric
  • Break metric down to provide relevance
  • Measure everything
  • Create dashboards to distill data
  • Make the data visible to everyone
  • Regular meetings (short and often) to create common language
  • Try running some A/B tests
  • Keep changes small and regular
  • Communicate, communicate, communicate
  • Enjoy your successes

Further Reading

This blog post was picked up by The Media Briefing who further interviewed myself and Jamie Walters for a slightly different angle on the above. The post How the Metro uses metrics to create flexible and effective digital products is well worth a read.

Music to write this code to

Nice bit of LTJ Bukem style drum and bass to get your head nodding to a new process.

 

How to setup an AWS EC2 instance to reduce WordPress hacker attacks

Hacker Terminal Window
WordPress hacker attacks have been on the rise this year and although I haven’t been hacked (touch wood) they have managed to overload my site on a number of occasions. I have been on a quest to stop them getting in and also to reduce the instances of their repeated entry attempts taking it down. This has been a pretty long journey and one that I thought other people might benefit from so I have decided to share it here.

Monitoring

The first step that I took was setting up alerts in the Amazon EC2 console to send me an email if my CPU stays at 100% for over five minutes. You can do this by clicking on your server in the console and then hitting the “Monitoring” tab and then “Create Alarm”. This seemed to be about the right amount of time to stop too many false positives and usually indicated that something was up with the server.

The second set of monitoring I setup was New Relic, we use them extensively at work to monitor our servers and their free version gives you a load of really useful information about how both the server and PHP are performing. Think of it as a version of Google Real Time that allows you to look at the processes and speed of everything going on inside your server and application. They also have a really useful availability monitoring service built in which you can setup to ping your site every few minutes to see if it is accessible and a great iPhone App that sends push notifications when something goes down.

Logs

The next thing you need to do is SSH into your server and check your httpd access_log to see what is trying to access your server.

sudo tail -f /etc/httpd/logs/access_log

This will bring up what is currently happening on your server and you can see all of the requests that are coming in. If your site does go down this is the place to check right afterwards to see what the requests were that overloaded it.

Mod Security

I decided to install Mod Security along with the OWASP ruleset to give myself an added extra layer of protection from any issues that might get introduced into WordPress before a patch comes out. I have to say that this involved quite a steep learning curve and due to the amount of technical information that I write in my blog so it took me a long time to modify all of the rules to allow saving of posts amongst other things.

As I am running on the AWS Centos image and it already had the EPEL repository setup so all I needed to do was:

sudo yum install mod_security mod_security_crs
sudo service httpd restart

Once this has installed you should definitely start clicking around your blog and see what is broken, quite likely that actions like saving a post, updating a comments status and updating plugins will now throw 403 errors. In order to allow your site to function need to create a file called whitelist.conf in the following location.

/etc/httpd/modsecurity.d/activated_rules

You then need to populate it with rules that allow you to bypass the specific rules that are triggered when using WordPress. Mod_security isn’t aware of your logged in status and assumes every call is external so this throws up quite a few issues. I have included a link to the ruleset that I have built up over the last few months below. There are probably more entries than most people will need but due to the geeky and code based nature of my posts a lot of stuff gets blocked, I have also enabled XMLRPC so apps work as well.

whitelist.conf

Adding your own rules to whitelist.conf

Should you get a 403 error when performing any action on your site you should SSH to your server and via the terminal type in the below.

sudo tail -f /etc/httpd/logs/modsec_audit.log

Then when you retry the action you should see another entry in the log and you need to pick out the data that looks like this.

[id "960035"]

You then just need to add the following line to your whitelist.conf

  SecRuleRemoveById 960035

Under the correct Location e.g.

<LocationMatch "/xmlrpc.php">
  SecRuleRemoveById 981173
</LocationMatch>

This can be quite a laborious process and possibly the whole lot is overkill but have to say I have learnt quite a bit in the process. I also added the following to the bottom of my http.conf which bans an IP from attempting to login for five minutes after three failed attempts.

# Stop Brute Force Attack by banning IP http://www.frameloss.org/2011/07/29/stopping-brute-force-logins-against-wordpress/

        # This has to be global, cannot exist within a directory or location clause . . .
        SecAction phase:1,nolog,pass,initcol:ip=%{REMOTE_ADDR},initcol:user=%{REMOTE_ADDR},id:2323

                # Setup brute force detection. 

                # React if block flag has been set.
                SecRule user:bf_block "@gt 0" "deny,status:401,log,msg:'ip address blocked for 5 minutes, more than 3 login attempts in 3 minutes.',id:2323"

                # Setup Tracking.  On a successful login, a 302 redirect is performed, a 200 indicates login failed.
                SecRule RESPONSE_STATUS "^302" "phase:5,t:none,nolog,pass,setvar:ip.bf_counter=0,id:2324"
                SecRule RESPONSE_STATUS "^200" "phase:5,chain,t:none,nolog,pass,setvar:ip.bf_counter=+1,deprecatevar:ip.bf_counter=1/180,id:2325"
                SecRule ip:bf_counter "@gt 3" "t:none,setvar:user.bf_block=1,expirevar:user.bf_block=300,setvar:ip.bf_counter=0"

htaccess

The final part of this is the .htaccess changes that I made to stop people being able to post without a referrer as a lot of bots use this method as they are computer programs rather than real people. I also had two specific User Agents that had targeted me which I block all access to my wp-login page.

# Stop protected folders from being narked. Also helps with spammers
ErrorDocument 401 /401.html

# Stop spam attack logins and comments
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteCond %{REQUEST_METHOD} POST
RewriteCond %{REQUEST_URI} .(wp-comments-post|wp-login)\.php*
RewriteCond %{HTTP_REFERER} !.*(blog.david-jensen.com|dev.mariamjensen.com|www.cafeontherye.co.uk).* [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(Windows\ NT\ 6\.1;\ WOW64\;\ rv\:18\.0\)\ Gecko/20100101\ Firefox/18\.0 [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mozilla/5\.0\ \(Parsley\ NT\ 1.0\;\ rv\:1.0\)\ Parsley/1.0.0.0 [OR]
RewriteCond %{HTTP_USER_AGENT} ^$
RewriteRule .* - [F]
</ifModule>

Here are a few sites that helped the creation of this post.

Definitely interested in hearing if anyone else has any other good ideas around how to approach this issue or has any tips from me as I am certainly no security expert. However after the uptime for my server has definitely improved over the last while with all of this helping to reduce WordPress hacker attacks on this blog.

Music to write this Code to

Andy C delivers some seriously high paced hacker music to help you get through this code in a jiffy.

22 tips for managing successful Agile projects

listen-to-this-wall

In the past year I have run two large successful agile projects that have delivered software using the Scrum methodology to build a minimum viable product (MVP) and then transitioned to Kanban once they have gone live. The first one was the Metro.co.uk responsive redesign and CMS migration which was delivered on time and under budget. Since going live it has been quickly iterated to deliver solid mobile and overall traffic growth for Metro. The second project is the Metro Play Casino which delivered its MVP as part of a large program of work that required us to get a fully regulated offshore gambling business started. This is now ahead of budgeted expectations and we are currently scoping the next phase of development. During both of these projects there have been some ups and downs but overall it has been a really positive experience. I thought I would share some tips that I have collated whilst managing these projects successfully using agile methodologies.

  1. Initially just define a few clear high level goals to measure success that allow detail to be filled in at the appropriate time.
  2. Start prototyping as early as possible, learning by doing is the most effective way to figure out what to do and also what not to do. Being able to actually use something as early as possible is very important. Even if this means releasing something different than your end goal.
  3. Don’t break down every story in the beginning and add it up to find out how long the project is going to take, this will just leave you with an unmanageable backlog and most of the stories will be out of date by the time you come to develop them. We have found that the best approach is to get a group of experts in the room and play planning poker for the project as Dan North suggests calling it Blink Estimation.
  4. Don’t be afraid to pivot your MVP or technology choices if the learning that you gather points in slightly different direction than you first thought. Knowing what you don’t want is as important as knowing what you want.
  5. Clear and consistent goals facilitate quicker decision making. These enable decisions to be made based on logic and this decisioning can be distributed around the team and post validated. Any single point of decision making is going to really slow things down, especially important in the last couple of iterations.
  6. If you are still developing a lot of new functionality in the last sprint be worried. Also leave a little bit at the end to tie it all together as things always pop up that you didn’t plan for.
  7. A more traditional Project Management layer can really help with external vendors and internal communication. Keep project plans at the highest possible level for visibility and better decision making especially on dependencies.
  8. Change your process, change it again. Retrospectives are really key to this process so don’t forget to have them after every sprint. Be agile about your process.
  9. Quite often order comes from short periods of carefully orchestrated chaos, these can help teams form by giving people common frustrations. Listening to these and then adapting your process to remove them helps the team to feel empowered.
  10. Get onto your production environment as soon as possible, no matter how similar other environments are they are never the same and issues will come out at this stage.
  11. Get developers and the business talking directly if at all possible. Learning together through structured conversation delivers the best business value. These can be facilitated by a Business Analyst or Product Owner but they should not always be in the middle as the lack of knowledge transfer will harm the project. This is especially true when dealing with multiple stakeholders.
  12. Bite off the biggest unknowns first and work on them to make them knowns. The best way to do this is to release something early and get real people to use your product. This doesn’t need to be MVP but a Minimum Learning Product that delivers some value to end users.
  13. You can assemble a group of people but it takes time and effort to form a team. Setting achievable goals and then ensuring they are delivered with a sprinkling of pressure early is a great way to prepare for the final few sprints.
  14. Restrictions bring out peoples creativity.
  15. In my opinion design should be emergent as should architecture, initially go with the most established patterns however you need to give the team time to fix things if they turn out to no longer be the right choice. This allows you to focus on actual problems rather than solving ones you only think you have.
  16. Automated testing should focus on smoke tests and high level indicators when things are rapidly changing.
  17. Automate builds and deployments in phase 0 for maximum payback.
  18. Constantly engage the business to make them define the problems they are trying to solve and then work together on the right solution for the project and the business in iterative steps.
  19. If you are going to take any risks with the project take those decisions at the latest possible point.
  20. Celebrate the team and their successes.
  21. Things will change constantly so learn to embrace it and hopefully get to the place where you enjoy it. If you have problems detailed rather than solutions then these can then be adapted as you learn.
  22. Simple things you haven’t done before are complicated to execute. Do not under estimate… or fall into the planning fallacy. Build complexity in over time.

Conclusion

Managing successful agile projects is something that can deliver serious business benefit if done with the right approach. The constant feedback that is available allows iterative learning and if your environment is setup to handle this will allow you to be very nimble. Quite often agile is an approach amongst a sea of different project management methodologies and I believe that if adapted properly to your environment can bring real benefits to most projects.

Music to write this Code to

Just Be (Bushwhaka!) delivers some serious house sounds in San Francisco making me wish it was sunny every time I listen.

Vrsus: “The future of real time voting” – DMGT Hackathon 2013

 

DMGT Hackathon

About six months ago I was sitting in a CTO meeting across DMGT and they were trying to come up with ways that we could improve collaboration across the group. After a few suggestions around wiki based API pages I came up with the idea to run a Hackathon. In my experience the best way to get geeks to know each other is let them build stuff together. This idea stuck and luckily Oliver Parke had sometime on his hands and got the job of organising it all. We were very fortunate to be able to get the Campus London Workspace to hold the event which is a hot bed of startups and geekery. The event was run on the 23rd and 24th of May and included technical people from businesses across DMGT including:

  • Metro
  • Mail Online
  • Euromoney
  • dmg::media Sprint Suite / Dev Lab
  • Landmark
  • Evenbase
  • Broadbean
  • Wowcher

Haribo Fueled Hacking

I was fortunate enough to also be able to participate in Haribo fuelled hacking and below is an overview of what we built and how we approached the whole event.

Vrsus: The future of real time voting

Trying to find a group in the initial hustle of the team forming stage was easier than I thought. My initial idea was something based around real time analytics but this quickly morphed into a real time voting tool to decide which the best cats on the internet are. The great thing about the real time approach was a chance for us to use a load of new technologies. So the dream team that we ended up with consisted of the following:

  • Mary Morgan (Euromoney Institutional Investor)
  • Frédéric Occédat (Mail Online)
  • Panagiotis Albanis (Metro)
  • George Old (dmg::media)
  • Me (Metro)

Once we had a general direction with regards to online voting we had an hour design session around our flip-board to try to conceptualise the flow through our app and what screens and actions would be necessary. This gave us a decent enough understanding of the requirements to put together a technology stack which consisted of the following tiers:

As we wanted to be real-time and socket based most of the stack chose itself and we thought that Redis would provide a simple way of storing the data and retrieving the post via score. At this stage we started to get down to the detail of building the initial flows around being able to submit images for voting on.

us vrsus them new post

I also started work on setting up the EC2 instance so we could easily pull from GitHub and keep the server running. There were a few things that I need to do in order to get the ports forwarded to HTTP and the Node process running in the background for which I used the following:

  • Forever – makes Node.js run as a background process
  • Rinetd – allows port forwarding on Linux so Node can run on port 80 and sockets on 8080

Next steps were to get all the pages setup and the logo done which took the rest of the evening.

Coming in the next day first thing we sorted was our Kanban board to get everyone working on the different bits that were required. This worked well through out the day as we finished the following screens.

you vrsus me home

you vrsus me category

us vrsus them sub-category

us vrsus them sub-category

you vrsus me vote

I also managed to procure the domain name vrsus.me and then added the subdomain you.vrsus.me to it to give it a bit of entertainment. The second day disappeared far to fast and I focused on stripping down the homepage to be nice and simple and also on the create page the ability to preview the image as soon as it has been added using sockets. We decided to against our idea of being able to enter any URL and for us to go and scrape the Open Graph or Twitter tags from it to get the images from the page to just entering a URL of an image.

This proved to be a good idea as it took us the rest of the day just to get the basic flow working, due to Redis not having any relational abilities there it took a bit of time to figure out how to store both the complex hash around the upload as well as the score after voting. The ability for the list to change order in real time also came right down to the wire and although there isn’t a fancy animation the list view is socket based and will change order when votes come in which I think adds a lot to the application.

It was then time to give a presentation of how we had approached everything and I had managed to knock up the below in Prezi.

We won the ‘Most Technically Impressive’ category overall.

Future Ideas

If we were to have some more time to develop this application into something more fully fledged I think these are the area’s that we would focus on.

  • Ability to post any url and vote on it
  • Social integrations
  • Ability to put sponsored posts in it
  • General responsive and cross browser performance
  • Improvements to the animation
  • Tracking

I think that Vrsus could be utilised across DMGT in many different ways apart from just showing the best cat beards on the internet. Having a real time voting system that people could submit content to that allowed the best to rise to the top based on user interaction would be a valuable asset in my opinion.

If you are interested in having a look at the code you can see it on GitHub and the url you.vrsus.me is still live and will be so for the next month or so.

Video

DMGT Hackathon – Google Campus from Dan Buck Joyce – OFILMS on Vimeo.

Photos

Vs Logo

Changing the Paradigm with Swipe

I was very fortunate to attend the 2013 WordPress VIP Developer conference in Napa Valley this year. It was great to meet all of the people who use WordPress at scale across the media space and the lovely people behind it all at Automattic. Also crazy to see that the average development team size for someone on WordPress VIP is 2-3, considering many of these sites are serving millions of uniques a day that is incredibly lean.

I was asked to give a lightening talk (less than 5 minutes) on a subject and I decided I would give a quick overview of how we have approached building Swipe into metro.co.uk. This seemed to go down quite well though qz.com were on after me and what they have done with their front end is even more impressive. I thought it would be a nice idea to put my slide deck up somewhere for viewing.

UPDATE: Metro have decided to remove swipe from Metro.co.uk for these reasons.

One of the other talks the previous day was by Yuri Victor on why the Washington Post uses WordPress and if you haven’t already seem it then it is definitely worth the visit even for the typography alone.

Other great notes from the few days were to use the following for WordPress debugging.

Also use Grunt to automate the use of these two libraries for image optimisation.

Generating fonts to help reduce images/sprites can be a good idea or just use Genericons which have been done for you. Also I found out that script tags on a page are completely blocking.

How to use Atlassian Jira with Greenhopper to manage multiple Backlogs

At Metro our development project has multiple product owners who each have their own backlog and want to prioritise them separately using Greenhopper’s drag and drop interface. We didn’t want to create multiple projects to enable this and Greenhopper’s multiple backlogs support for a single project is something that had previously eluded me. Ade Shokoya who was our business analyst came up with the idea of creating an additional Kanban rapid board with product owner based swimlanes to achieve this. We now have one Kanban rapid board to manage the product owner backlogs and one to manage development priorities. They share a column which allow tasks to flow easily from one to the other.

We use components within our Jira project to categorise tasks by the business area and product owner that they deliver value for. Each component allows you to setup a default owner and this ensures that they are notified when any tickets are added or amended. We created an additional Kanban rapid board specifically for backlog management and added “Backlog”, “To Do” and “Scheduled for Development” columns to this. We then enabled swimlanes based on components so each of the individual backlogs can be prioritised separately within these. We also utilise quick filters based on component to allow each product owner to only see their tickets on the rapid board. As the filters are URL based this allows each product owner to have a browser bookmark to take them straight to their backlog.

Setup Swimlanes per Component

Greenhopper Multiple Backlogs Rapid Board Swimlanes

Setup Quick Filter per Component

Greenhopper Multiple Backlogs Rapid Board Quick Filters

Greenhopper Multiple Backlogs Kanban Rapid Board with Swimlanes and Quick Filters

Greenhopper Multiple Backlog Product Owner Board

Getting the business engaged with development process and priorities in Jira is a long term project of mine (as otherwise I am the only person who manages the board). The best way that I have found to do this is hold specific product owner stand ups twice a week where we have a look at the work in progress and discuss what the priorities are for the items about to move into the development stream. We have this around the TV that has our development Kanban rapid board displayed on it and can quickly switch between that and the backlog view. This allows each of the product owners to get a good view of overall priorities and where their requests fit into that. They also see how the process works and have an opportunity to provide feedback. This is complimented by a meeting every other week where we look at how each idea is delivering value based on performance data and then decide whether we do more or less of this based on our overall strategy.

Having multiple Rapid Boards for a more complicated flow that keeps each of the boards simple is one of the best moves that we have made. We are also now using a separate board to tightly manage our testing and release process as we have multiple environments and stages that overcomplicate the main board. Greenhopper has been really good at adding new features and streamlining their existing ones to provide a great way of managing the different stages of the project in a single dashboard.

Metro.co.uk Greenhopper Kanban Rapid Board Dev Board

Metro Greenhopper Development Board

Background

I have been using the Atlassian stack including Jira and Greenhopper to manage multiple Agile projects using its Scrum templates over the last three years. The great thing about Jira (and at times the most frustrating thing) is that it is pretty much infinitely configurable and Greenhopper gives a great drag and drop interface for Scrum and Kanban setups. Personally I don’t have the time to manage both an online system and a paper based post it system so we moved everything into Jira and got a 42″ television which is used for displaying boards during standups and as an information radiator during the day.

During the minimum viable product (MVP) phase of the metro.co.uk CMS migration we had a nice simple setup of Greenhopper utilising its Scrum template with a dedicated Business Analyst (BA) who managed the backlog. However he was a project resource and his contract was about to come to an end, we had also shifted from using Scurm to using Kanban but were initially struggling with the flow from the large backlog left over from the project.

Music to write this Code to

Treasure Fingers once again giving away a sparkling mix that makes you wish it was summer all year round.