Sunday, March 23, 2014

Radio NZ Browser Stats - March 2014

Here is another lot of browser and OS stats for www.radionz.co.nz and The Wireless.

Size of Content Library

The audio library now contains 27,000 hours of material, all of it searchable from www.radionz.co.nz/audio. That is 179,000 items, most of which are downloadable and embeddable.

We also just partnered with DigitalNZ who now have a copy of most of our audio metadata, and update it each night with new content.

Browsers


Browser 03/2014 11/201306/201211/201111/201011/200911/2008
Chrome 25.23 23.7 14.6 13.8 8.75 4.2 1.47
IE 24.02 26.337.541.250.65663
Safari 18.94 17.5217.35.613.1105.66
Firefox 14.81 16.619.823.225.5227.527.73
Android 11.71 117.5
Opera 0.46 0.690.70.91.021.08

IE is still in decline, and IE6 is 0.32%, IE7 1.86%, IE8 25%, IE9 20.7%, IE10 13%. IE11 is new at 37%.

At The Wireless things are very different: Chrome 37%, Safari (in app) 20%, Firefox 13%, Safari 12.5%, IE 11.3%, Android 5.0%

Operating System

OS 03/2014 11/201306/201211/201111/201011/200911/2008
Windows 52.9 5867.3728184.889.3
iOS 16.6 147.88
Android 16.5 13.77.793.60.30.020
Mac 10.7 13.414.715.614.212.68.5
iPhone 2.51.40.560.19
iPad 2.40.6300
Linux 1.9 1.81.271.531.41.451.72
Win Phone 0.44 0.39
iPod 0.50.350.220.08

And at The Wireless, Windows 42%, iOS 26.5%, Mac 18.3%, Android 11%, Linux .78%, Win Phone 0.48%

Mobile

In June 2012 mobile was 16% of our traffic. In November 2013 it was 29%. It is now 23.4% mobile and 10.8% for tablet, making 34.2%.

The most popular devices are the same as last time - iPad (25%), iPhone (22%), and Samsung GT-I9300 Galaxy S III (5%)

Over at The Wireless 38.3% of the traffic is mobile, with popular devices being iPhone (51%), iPad (17%) Galaxy S III (5%).

Saturday, November 23, 2013

November 2013 Browser Stats for Radio NZ

Here it is once again - my roundup of browser and OS stats for www.radionz.co.nz. This time I'm including stats from The Wireless.

Size of Content Library

The audio library now contains 25,000 hours of material, all of it searchable from www.radionz.co.nz/audio. That is 167,000 items, most of which are downloadable and embeddable.

Browsers

Browser 11/2013 06/201211/201111/201011/200911/2008
IE 26.3 37.541.250.65663
Chrome 23.7 14.6 13.8 8.75 4.2 1.47
Safari 17.52 17.3 5.6 13.1 10 5.66
Firefox 16.6 19.823.225.5227.527.73
Android 11 7.5
Opera 0.690.70.91.021.08

IE is in decline, and IE6 is 0.6%, IE7 3%, IE8 31%, IE9 22.6%, IE10 42%.

At The Wireless things are very different: Chrome 35%, Safari 16%, Safari (in app) 16%, Firefox 15%, IE 9.4%, Android 5.5%

Operating System

OS 11/2013 06/201211/201111/201011/200911/2008
Windows 58 67.3728184.889.3
iOS 14 7.88
Android 13.7 7.793.60.30.020
Mac 13.4 14.7 15.6 14.2 12.6 8.5
iPhone 2.51.40.560.19
iPad 2.40.6300
Linux 1.8 1.271.531.41.451.72
Win Phone 0.39
iPod 0.50.350.220.08

And at The Wireless, Windows 40%, iOS 26%, Mac 20%, Android 10%, Linux 1.9%, Win Phone 0.55%

Mobile

When I last published stats in June 2012 mobile was 16% of our traffic. It is now 29%.

The most popular devices are iPad (27%), iPhone (21%), and Samsung GT-I9300 Galaxy S III (6%)

Over at The Wireless 36% of the traffic is mobile, with popular devices being iPhone (47%), iPad (21%) Galaxy S III (4%).

Social

20% of referrals to radionz.co.nz come from social sites (and 35% from Google news).

73% of referrals to thewireless.co.nz come from social sites.

Saturday, June 8, 2013

Why you should allow time (and pay) for refactoring of code

At Radio NZ we engage outside contractors to help us build the web applications we use. The most public of these apps is our website, and internally there is a large database system that is used company-wide.

One of the challenges of writing code is that over time the codebase tends to become rundown. This can be made worse if many people are working on that code. Differences in coding styles, changes in best-practices over time, current (and past) fashion; these all contribute to software rot.

That is not to say that the code was bad when it was first written, or that the original developer did not know what they were doing. Most likely, they did know what they were doing at the time, based on what they knew at that time. Perhaps there were other contraints (such as limited time). Someone did there best, at the time.

Refactoring is a critically important part of maintaining a software application.

Wikipedia describes refactoring:
Code refactoring is a "disciplined technique for restructuring an existing body of code, altering its internal structure without changing its external behavior", undertaken in order to improve some of the nonfunctional attributes of the software. Advantages include improved code readability and reduced complexity to improve the maintainability of the source code, as well as a more expressive internal architecture or object model to improve extensibility.
It goes on to quote Joshua Kerievsky:
By continuously improving the design of code, we make it easier and easier to work with. This is in sharp contrast to what typically happens: little refactoring and a great deal of attention paid to expediently adding new features. If you get into the hygienic habit of refactoring continuously, you'll find that it is easier to extend and maintain code. — Joshua Kerievsky, Refactoring to Patterns

Some will say, "if it ain't broke don't fix it".

Apply that to your car or your house and gardens and see where it gets you. Where it gets you is: broken down in the middle of nowhere, and your phone battery is flat because who needs to charge those things anyway. Or a house that has to be re-clad because you did not repaint it early enough, and the garden, well, I think we have all seen those.

Your code is a valuable asset. It contains business logic - knowledge about how things should work that have been built up over time, and codified in the software. You paid for the process that converted that knowledge to code.

The source code repository also contains (or should contain) business intelligence about why certain decisions were made, and the evolution of the code over time. This is vital information for future developers. If you do not have a code repository, please look for another field of work. Now.

Refactoring has a cost attached. It takes time. But at its core, refactoring is about reducing future costs.

You must accept the cost of refactoring, and make time for your developers to do this work.

The benefits are twofold:

1. Better quality code that costs less to maintain over time.

2. Your staff will be more productive.

On point two, everyone wants to have pride in their work. For programmers, refactoring code is a key part of the process of remaining connected to the code, and experiencing joy in their work. Many studies have shown that happy staff are more productive, on top of the gains of working with quality code.

At Radio NZ I encourage the developers who work for us to suggest refactorings, and I allocate the time for them to do the work because I see this as an investment in the future. The person who has to maintain this code might be me. Or it might be you

Forcing people to continue to work on code they know could be improved without letting them make those improvements is bad management.

Some of the code in your system will be hard to refactor. It is probable that this code was put together quickly, possibly without tests. Don't let this put you off. Read and understand the concept of technical debt.

Also, watch this excellent presentation where Katrina Owen refactors some code that has no tests. Watch it a second time if you don't quite get it. If you are technical, read Martin Folwer's book.

If you use Ruby and/or Rails read the book Practical Object-Oriented Design in Ruby. The approach outlined in book has fuelled many refactorings in our projects.

On point about tests. Having good tests is necessary to facilitate refactoring. Without tests you have no way to know for sure that the refactored code behaves the same as the original code.

Chances are, if you don't already have good tests, you will already being seeing application errors after changes to the code, and that these will be hard to find and fix. You'll also see new bugs appear as the direct result of fixing an existing bug. Yes, just like Novapay.

I suggest you do three things:

1. Right now, go and ask your key developers, is there some part of the application that is wasting their time and should be refactored. Get an estimate of how much time is required, and schedule it.

2. Start a discussion with your developers about the overal design of the app. Does it support where the business is now? Is it making future enhancements difficult, or reducing productivity and/or job satisfaction?

3. And more broadly, find out from from each of your developers three things that they feel make it hard for them to produce quality work. Don't put your own definition on quality, just ask the question. Make a commitment to fix these things.

Sunday, May 26, 2013

The New Radio New Zealand Website

After three years of work, the new Radio New Zealand website is about to go now live.

As well as a new design, we built a new content management system to handle our publishing requirements.

Starting from the inside out, this is what our technology stack looks like.

The Hardware

We have eight HP servers, all with 8-core hyper-threading CPUs, and a minimum of 12 gigs of RAM. Four of these are at our Wellington office, and four are at our Auckland office. We'd previously hosted five machines at ICONZ data centre in Auckland.

Each group of machines is connected to the internet via a Juniper Firewall with dual edge-routers (for redundancy) connecting to FX Networks via Gigabit fibre. And yes, we can run both sites at 1 Gigabit if we need to.

Self-hosting was, as they say, a no-brainer. We have a server room at both locations, each with UPS and on-site generator. The deal we have with FX provides price certainty under a range of extreme scenarios. For example, the traffic we experienced after the Christchurch earthquake in February 2011 was 50 times higher than normal and stayed at 15% higher for days afterwards. We already move about 500 Gigabytes of data a day, so large peaks in traffic can cause large peaks in cost.

The dual site approach allows for us to continue operating if one site is cut off for some reason, and to share load between them if required.

We did consider hosting in the cloud, but two things counted against this approach. The first is we had eight high-end servers with 3 years depreciation still to run, and secondly the cost of provisioning for peaky and unpredictable traffic patterns. Our strong preference is to host in New Zealand to give local visitors the fastest possible service, and the price-point of local cloud services is not quite there for an operation of our size.

The Operating Systems

Our primary operating system (on 6 servers) is GNU/Debian Linux. All run as bare-metal with virtual machines (VM) running on top. VMs can be moved between cities with almost no gap in service.

We also have a pair of Windows Media Servers (one in each town) to handle live streaming and on-demand audio, but these will be retired later in the year and replaced with a Wowza streaming server. That change will give us much better cross-platform streaming capability.

The whole cluster was set up by Simon Blake of Katipo.

The Content Management System

Our content management system (CMS) is called ELF, and was built using the Ruby On Rails framework. We chose Rails because it is used internally for some of our intranet applications, and because it was a good fit for type of content we offer.

This replaces MySource Matrix, which we used from 2005 until we started phasing it out in 2010.

The first code for ELF was written on 8 March 2010, and in the month that followed Nigel Ramsay and Marcus Baguley from AbleTech wrote a proof on concept based around the recipes section of the site.

I was learning Rails myself at this point, from a background in 8086 Assembly, C, Perl and PHP. Later in the project we used external contractors for larger blocks of work and what I call 'heavy lifting' - complex features that require extended periods of work. Shevaun Coker has done much of this work and was joined by Cameron Prebble recently (both from AbleTech).

A proof of concept was used to assess the search tool (Apache Solr) and load profile of the system (how many pages it could serve). This was successful, with the platform being able to serve around 850 requests a second. This compared very well to the 20-30 requests a second upper limit of the old system.

Over the following 18 months large sections of the site were replaced, with news (the largest) replaced just weeks before the 22 February earthquake.

The last major section of Matrix was replaced late last year. In all, several hundred thousand pieces of content were migrated from Matrix to ELF.

The administration section of ELF has been written specifically to cater to the needs of busy broadcast producers and is highly optimised for the work that we do. We have built only the features that we need, avoiding clutter from features we don't want or use.

A producer can learn all they need to know about publishing their content in about 10 minutes.

ELF is about 126,000 lines of code, and the test suite has 2,200 tests that help us ensure the system remains free of bugs. A recent check of the system found that on average vistors to the site got a system error (a 500 error page) once every 40,000 page views. All bugs get attended to, so that number is now even lower.

We use a Varnish cache in front of the Rails application server, and they work in concert to deliver pages as quickly as possible, even when there is high demand.

There is a very detailed series of posts about the design and building of ELF, and the migration of content, right here on this blog. The series is called Rebuilding Radio NZ.

Publishing Systems

We publish content in three main ways:
  • manually, via the administration interface of ELF
  • from our news editing system (iNews);
  • and from our on-air audio system (CoSTAR)
I have written software that allows our text and audio system to publish directly into the CMS. This enables staff to use their regular tools to publish, and no knowledge of HTML is required.

News content takes 15 seconds to update once the publish key is pressed, while audio takes a little longer due to the need to encode it for web delivery. We typically can have a piece of audio on the web a few minutes after broadcast.

Pre-recorded programmes (such as Our Changing World, Insight, Spectrum and many others) are pre-published, with content going live automatically after broadcast.

All these processes have been automated wherever possible, and they are highly optimised for speed, flexibility and reliability.

The Design

The aim was to simplify the site and let the content shine through.

We have retained many familiar elements in the design, improved others, and thrown away stuff that did not work. In the last three months there has been a vast amount of polishing work done - selecting great images, taking new staff photos, and reviewing all the content. It's been a long process!

Two years ago I wrote a post about dealing with doubt, and I wondered if we'd ever finish the project. I think if I'd known it would take another two years I might have given up then. Part of the problem was that a number of large projects I was working on got delayed, and I ended up working on them all at the same time.

Now we're done with this phase of the project, I am more than satisfied that the journey has been worth the trouble. We have a great new design, a CMS that'll cope with all our needs, and can continue to grow and be modified as we evolve the site.

The next phase is....

...you'll have to wait and see!

The Credits

My colleagues Frances Hopkins and Helena Nimmo deserve special mention. Their commitment and drive has ensured that the finished site has reached a level that is much, much more that the sum of its parts.

We've been supported by a great group of contractors (in no particular order): Shevaun Coker, Nigel Ramsay, Cameron Prebble, Marcus Baguley, Michael Koziarski, Alison Green, Susannah 'Roxy' Field, David Buck, Simon Blake, Amnon Ben-Or, Amanda Dorrell, John Moore, Phillipa Devenport-Johns, and Emma Harrison.

Feedback or comment about the the site can be sent to rnzwebsite@radionz.co.nz.

I am happy to answer any technical questions in the comments.

Sunday, February 10, 2013

Radio NZ stats February 2013

As anyone who hasn't been living under a rock for the last few years will know, there has been an explosion of the use of mobile computing devices over the last few years. At Radio New Zealand we've seen the percentage of mobiles devices accessing our website increase dramatically in the last 12 months.

One year ago the percentage of our traffic from mobile devices (including tablets) was 13%. This month it is 27%.

The type of devices has also shifted. A year ago iDevices and Android were neck and neck. This month devices powered by Android are the clear leader:

OSPerc.
Android59.77
iOS38.79
Windows Phone0.58
Blackberry0.49
Symbian0.15
Nokia0.07
Samsung0.05
Windows0.05
Sony0.04

I expect this trend to continue, but it is impossible to say how much further it'll go.

Browser use is also shifting. A year ago IE was at 40% and Firefox was at 21%. Safari was 16% and Chrome 12.76. The Android browser was 6%.

This month's figures:

BrowserPerc.
IE30
Safari17.4
Firefox16.7
Chrome16.3
Android Browser15.6
Safari (in app)2
Opera0.57
Opera Mini0.31

There has also been a big change in operating systems:

OS12 months agoLast month
Windows69.7%57%
Macintosh14.9%12.5%
Android6.19%17.5%
iPhone/iPad6.4%10.9%
Linux1.37%1.24%
Windows Phone00.16%

If you are are still designing sites for one browser or OS, now might be the time to change your strategy.