I still remember back when the Internet was an escape from the real world. Today, it seems that the real world is an escape from the Internet. Over the last 30 years, a substantial amount of daily life has moved online for billions of people: News, communication, sharing, commerce, banking, and entertainment.
Wherever human activity goes, regulation inevitably follows. This year it seems like the internet is truly going mainstream, with new regulations from 2 of the 3 largest economies in the world: the US’ FOSTA, and Europe’s GDPR.
I’m actually quite a fan of GDPR – or at least, what it’s trying to do. In GDPR, there’s an attempt to extend protection of EU citizens rights across the Internet, following their personal data wherever it may end up. And the regulations make a lot of sense from a consumer perspective: If companies want your personal data, they have to prove they can handle it responsibly, and you retain basically all the rights.
GDPR goes “live” on the 25th of May (just under 3 weeks!), and I’m hoping it heralds a new era of more responsible data practices. I think everything they’re trying to do is achievable, and should be baked into organizational and system design for new businesses (“privacy-first architecture”).
For existing businesses though, it’s going to be hell of a slog. I’ve worked on data audits and compliance for some of the world’s biggest brands, and it’s impossible to overstate how tricky full compliance is going to be. When your database software vendors have gone defunct, when your processes have spiraled out into spreadsheet nightmares, when your primary method of data exchange is email attachments, you’ve got a problem.
Over the last few days I’ve put some stuff live to help with that. For one, there’s already a wave of companies that are blocking the EU from accessing their services outright. It’s a sensible short-term move if you need time to assess the impact, and update your processes to comply. The rights of Erasure and Portability could require non-trivial work if you’ve been historically lax with how you manage data internally.
For that use-case, I pulled together a repo (gdpr-blackhole) of the IP ranges (IPv4 and IPv6) of all 28 EU member states, UK included. Brexit or not, early indications are that the UK will adopt GDPR in its entirety.
Then, since I work mainly in Laravel, I wrapped that up in a simple Middleware class that makes blocking EU IPs straightforward (laravel-gdpr-blocker).
Finally, I’ve just put a short blog post live on Amberstone: “What small businesses need to know about GDPR“. Large businesses already have armies of lawyers, auditors, and officers to assess their liability. Small businesses are not exempt, and if you want to participate in the second-largest economy on the planet, you’re going to need to know a bit about what GDPR asks of you.
Overall, it’s an exciting change. The ICO has already indicated that they’d sooner work with non-compliant organizations to improve their stewardship of personal data, making the eye-watering fines a last resort. And if all goes well, this is exactly the sort of bar-raising work we’ve needed on the world stage.
And with any luck, it means less Excel spreadsheets loaded with personal data. Time will tell 🙂
So you’re a South African business that wants to take payments for a subscription-based service. You’re probably going to have a mix of local and international customers, and want to know what services are available.
Well, I’ve got some good news, and some bad news.
The good news is – there are ways of doing this. Even with South Africa’s relative backwardness in global economic participation there are still a few options.
The bad news is: the best options are currently unavailable to you, and are likely to stay unavailable in the long term. If you have the means to do so, incorporating and setting up banking in a major economy might be the better option.
Local Options – EFT
It goes without saying that local EFT is an option. If you don’t want to wait for interbank delays, there are a few vendors that offer Instant EFT solutions:
In my experience, PayFast has the easier onboarding path, but you might have a tougher time integrating it into your application. None of those solutions incorporate any sort of subscription management though – it’s on you to keep the accounts in check.
PayFast allows you to accept credit cards online, and is relatively easy to set up if you’re not looking for advanced integration. They will make your life easier on one front – they understand subscription management: https://www.payfast.co.za/subscriptions/
I’ve worked with their API before though, and you’re going to need to exercise extreme patience with it in order to get anything done. I hope their systems and documentation improve over time!
There are too many global vendors to list, but when it comes to what you can feasibly use in South Africa, it narrows down to one pretty quickly: PayPal.
I implemented full-on subscription management using PayPal for Write500 – and it worked quite well. PayPal can accept credit cards, manage subscriptions (including pausing and resuming), and has a solid set of developer tools for integrating it into your application.
You’ll want to create a Billing Plan (your product), then a Billing Agreement (a paid subscription to it), so that Paypal can issue Invoices and settle them automatically. Start here: https://developer.paypal.com/docs/api
Going further abroad, you do have the option of incorporating a business remotely. It’s a ton of paperwork (depending on a ton of factors), but one vendor is offering a simple solution to it: Stripe Atlas.
For a one-time fee of $500, and a decent amount of effort, you can incorporate remotely in Delaware – including all the documentation and registration you need to run a corporation within the US. That includes a Stripe.com account, which is basically the global, golden standard for subscription credit card billing.
There’s a couple of major downsides on the administrative side, though – repatriating that money comes with its own set of tax challenges. From what I’ve seen, this is only a really viable option if you’re expecting to do a lot of subscription billing for global customers, and the Stripe fees are a more attractive option.
I would be remiss if I didn’t mention PeachPayments – a Cape Town-based company that attempts to make all of the above easier for local companies. If you’re not keen on the idea of getting your hands dirty with integration logic and setting up subscription billing, those will be the folks to talk to.
Then of course, there’s bitcoin. If you really feel like jumping into the murky waters of cryptocurrency for your project, give Coinbase a try. They offer a recurring billing option denominated in Bitcoin. To cash those coins out locally, the simplest option will probably be to hold a Luno.com wallet, and sell those coins on the local exchange to recover your Rands.
I know that there are likely several other services out there, but if I were starting from scratch today, and wanted to be able to accept payments from a global customer base I’d probably still go with PayPal, myself.
Right now, we’re living in one of the most momentous times in human history, and it could end up being one of the best (or, possibly, worst) things to unfold: our inevitable transition to what Maurice Conti calls the Augmented Age.
Computers have become part of mainstream life in every advanced economy, and basically all major cities around the world (into which people are packing in ever-greater numbers). The resulting efficiency gains have either been a huge boost to creativity and opportunity, or the death-knell of industries that employ tens of millions of people.
I’d like to share two different perspectives on this – both, conveniently, delivered as excellent TED talks. The first is by Maurice Conti, on how advances in computing have changed the way design could be done.
The most remarkable thing about the computer-derived inventions is how biological they look. It took nature millions of years to evolve a structure that their computers can do inside of a few days (referring to the drone chassis), and in future, could do on demand.
I think this is the best insight into how the leading edge of computing might change the way we design cities, vehicles, infrastructure, and the machines that help run our lives. It’s encouraging to note that human designers are still very much a part of the process, but will be able to do a lot more in a lot less time.
Which is a factor leading into the next TED talk – what happens when you centralize that amount of power (and consequently, the financial gains) in the hands of a relative few? People who are skilled at these technologies are able to create enormous value in a short space of time, relative to someone still doing the same task manually.
So what happens when you no longer have a need for the manual labor?
Another excellent talk that takes an unbiased view of Unconditional (I prefer Universal) Basic Income. It raises some good points, but misses at least one point I need to make a note of:
While it’s true that the top 5 tech companies are enormously valuable and employ relatively few people, the platforms they create have in turn generated opportunities for millions more. There are companies, products, services and entertainment channels that could not have existed were it not for the infrastructure and tools that Facebook and the like provide.
Google basically pulled the web development industry up out of the ground when it became clear to businesses that having a well-built site was a competitive advantage. I’m not sure anyone can count the amount of new jobs created in web development, creative design, copywriting, SEO optimization, consulting and education as a result of the platform Google built.
(Yes, I know Google didn’t build the internet. And yes, I know all these websites run on the internet that Google didn’t build, but everyone who’s ever been paid to build one has done so at the request of a customer who believed that being discoverable online would be beneficial to their business, and Google is still the king of discovery on the internet.)
Same goes for the use-cases enabled by Apple hardware, Facebook’s networking, Amazon’s fulfillment infrastructure, and the productivity tools released by Microsoft. Those companies themselves may employ relatively few, but they have empowered millions more.
I think UBI is feasible not so much because of productivity gains due to automation, but because of the ever-declining costs of providing an acceptable standard of living. An excellent, recent example of this is Apis Cor’s house printer.
On the one hand: This technology might end up putting a lot of construction workers out of jobs. While you’ll still need workers for big buildings and the like, simple 1-2 person houses can probably be built quickly, and very cheaply, as a result of this innovation.
But on the flip-side, the cost of houses will plummet. You may not need to work for 20 years to pay off a mortgage for a house that only costs $10k to build. While construction workers might be worried about this, the people who should be a lot more worried are ones with heavy investments in residential development companies 😉
I like to imagine a future unconstrained by urbanization. Cities are where the opportunities are – the best jobs are in cities, the best entertainment, the best healthcare, and overall, the best opportunities to live a good life. This is because it’s a lot easier, with the current limitations, to pile a lot of services into one place.
I don’t believe civilization needs to be so centralized, though. If you could get the same quality of food, healthcare, entertainment and job opportunity in an area 200km outside a major city, plus it was cheaper to live there – wouldn’t you?
And there may come a time when we have to. Most major cities (and by extension, most of the world’s population) are located relatively close to a coastline. Historically, cities were founded and grew near coastlines because those afforded the best opportunities for global trade.
Well, that’s under threat. Depending on who you believe, climate change is either a myth, or it’s a reality already underway – and one of the most dire consequences will be the rise of the ocean level. Which, if that happens, will start to make the large, coastal cities unlivable.
We will be forced to start again – massive inland migrations, the design of new cities, infrastructure and services to support the population, while simultaneously ensuring people have a shot at an acceptable standard of living. With the lessons we’re learning today, I imagine those cities (and societies) will look very different.
Between the work of engineers like Maurice and researchers like Federico, I’m optimistic that we’ll be well-equipped to meet those challenges in future.
Obvious upfront disclaimer: Don’t take financial advice from the internet, least of all me. And don’t sue me if you make a bad bet and lose – like I’ll point out below, this is still an extremely volatile space.
I have “mixed” opinions on cryptocurrencies. On the one hand, I’m enormously frustrated with the quasi-religious evangelism that tells people how crypto is the future of literally all money, and their pooh-poohing of simple technical concerns like “how do you scale to hundreds of millions of concurrent real-time transactions”.
Crypto as we know it today won’t replace money as we know it today. Some future optimized version of crypto might, but it will be unrecognizable as compared to the algorithms of today.
On other aspects, I’m a lot more positive – the technology behind it (blockchain) is the real revolution, and a completely new way of thinking about storing, processing and ensuring the integrity of data. New applications built on blockchain technology have enormous potential, and it’s all open-source.
The reason I put “mixed” in quotes is that these positions seem inconsistent to fans of crypto (“how can you not believe it’s the future of money if you think it’s so cool?”), which is an argument entirely on its own, and more than I want to get into right now.
So what are cryptocurrencies? As simply as I can explain it: digital gold.
A bit more complex: A shared ledger, secured by a mathematical algorithm, calculated across tens of thousands of computers which will reach consensus on whether or not a single transaction on it is authentic.
In other words – it’s impossible to forge, and it’s impossible to steal (short of actually forcing someone at gunpoint to transfer coins to you). It is possible to lose, however (as the upcoming BIP 148 hiccup might end up demonstrating), and of course it’s always possible that governments might decide to regulate or outlaw it altogether.
It’s also impossible to devalue by manipulating volume – each currency has a set upfront limit of how many coins will ever be issued, and free markets are entirely responsible for determining the value. I’m sure this is one of the more attractive aspects to the folks who deeply distrust any government, bank or other institution.
Which is where the volatility comes into it. It’s very unlikely, but it is possible at any point that one of these currencies (and there are many) could simply stop operating.
Their existence depends entirely on the network of miners (networked computers) that agree to process transactions for that currency, and at some point, regular external market forces could make some of the smaller currencies untenable.
There’s also the regulation aspect hanging over this. Apart from the obvious criminal uses of a nearly-untraceable cryptocurrency, there’s the wildly unregulated speculation going on right now.
As of today, Bitcoin alone has a USD-denominated market cap of over $40 billion. What this means in practical terms: A good couple of investors (in the hundreds-of-thousands range by now, at least) have bought Bitcoins in exchange for real money, likely in the hope that it’ll appreciate in value over time.
That’s a really big roulette wheel, and so far (for the most part) it’s paid off quite handsomely. It has a lot of the hallmarks of historical gold rushes: Someone’s found something of value, and people are tripping over each other to get in on the action.
That value, though, is likely to only ever exist in the realm of financial speculation. Bitcoin is a desirable, limited-edition asset, which in and of itself is enough to drive speculators and gamblers to it. At $40bn, you have to assume there’s a lot of interest in keeping the entire system propped up, almost entirely for its own sake.
Which is not necessarily a bad thing, and it’s why I’m optimistic about crypto. $40bn is technically a tiny market cap, compared to assets like gold ($7 trillion) – but it has proven itself over the last few years to be stable and growing.
And it can only grow. Sure, getting into Bitcoin today is hard. If you’re planning on buying hardware and setting up as a miner, you better hope you can get free electricity – otherwise you’re losing money the moment you turn that machine on.
The market can always expand, though, and this is the main reason for my optimism. Bitcoin is the largest, oldest and most well-known, but it’s by far not the only cryptocurrency that can be traded for fiat. Here’s the top 10 by market cap, as per coinmarketcap.com:
Each symbol there represents an entirely different blockchain – a different asset. And while a single currency has a set limit on the amount of coins it can produce, there’s no limit to the amount of currencies that can be set up.
Unlike gold, wine, art or land, cryptocurrencies can be forked and spun up infinitely. As an asset class, it can expand to soak up every bit of market demand. Every person in the world, theoretically, could have holdings in at least one cryptocurrency.
So why the title of this post, and why do I think it’s worth investing in? Simple: It’s still early days, and it’s still got lots of room to grow.
With the amount of interest and investment going into the underlying systems, and the value of owning some cryptocurrency (both inherent and perceived), they’re all likely to continue to grow in value over the long term.
Today, a single Bitcoin costs around $2’500. There’s no reason to believe it won’t hit $10’000 someday. The same is true of most of the currencies in that screenshot above – whatever the cost is today, there’s a good chance it’ll be a great deal higher in a few years.
My quick anecdote on this: In January I moved R800 into a local crypto exchange, then found out they wanted to do KYC processes over unencrypted email, got annoyed, asked for a refund, and was denied. So in frustration I just cut my losses and forgot about it.
Until a few days ago, when I remembered the whole saga and logged in to see if anything had changed. And one thing had: In around 5 months, my R800 investment in Litecoin had turned into R4’720.
That’s a return you’re not going to find anywhere else – which is what makes it simultaneously alluring, and incredibly risky. There could just as easily have been a crash that wiped out my R800 forever.
Which is why I started small, and would advise anyone else interested in this to do the same. I’ve got accounts on three different exchanges, and holdings in two different cryptocurrencies. Every now and then when I get a bit of spare cash left over that I can afford to save, I move some of it into (what I now consider) my cryptocurrency portfolio.
Between all that spare cash, accumulating over time, it’s added up to very decent growth. As of today, that rate is sitting around 35%, and I’ve been doing this less than a year. That’s an absolutely insane growth rate as compared to other investment options.
And it could all disappear tomorrow, leaving me with absolutely no recourse. Which is why, as tempting as it is sometimes, I’m not gonna bet the farm on this just yet.
What it is, though, is the single riskiest position across all my investments, and the one that’s most likely to yield big returns in a short space of time. I’m expecting (and somewhat hoping) that this growth trend will continue for the next few years, too.
But I’m also expecting that at some point, all the valuations might just evaporate overnight, and I would have done no better than slowly setting that cash on fire. Tempered expectations are pretty much a requirement if you’re gonna ride this roller-coaster – at least, in my experience!
Howdy, stranger! This document is the other half of this video, in which I set up a single-server instance of Mastodon. This was assembled on 9 April 2017, and there’s a good chance that some of the specifics here will change over time. I’ll keep an updated version up on wogan.blog.
(What is Mastodon? I’ll do another post on that sometime!)
UPDATE 17 April 2017: Mastodon has reached v1.2, and now requires Ruby 2.4.1. The post has been updated with new commands required as of today, and an upgrade guide is below.
At a bare minimum, you’re going to need:
A domain name, with the ability to add an A record yourself
A free mailgun.com account, with the account verified and your sandbox enabled to send to you
A 1GB RAM machine with decent network access. This document uses a DigitalOcean VM.
This setup procedure skips a few things that you may want to do on a “productionized” or “community” instance of Mastodon, such as configuring S3 file storage, or using a non-sandbox email send account. You may also want a beefier machine than just 1GB RAM.
For reference, the OS version in use is Ubuntu 16.04.2 LTS and all the commands are being run from the root user unless explicitly specified.
1. Getting started!
The first couple steps:
Create the VM
Point your domain to it immediately, by setting the A record to the public IP
Log into the VM
Set your root password
Create a new Mastodon user: adduser mastodon
Update the apt cache: apt-get update
2. Install Prerequisites
Now we’ll grab all the prerequisite software packages in one go:
You’ll also want to be sure that redis is running, so do:
# service redis-server start
3. Configure Database
With Postgres installed, you need to create a new user. Drop into the postgres user and create a mastodon account:
# su - postgres
> CREATE USER mastodon CREATEDB;
Later on we’ll configure mastodon to use that.
4. Generate SSL certificate
Before configuring nginx, we can generate the files we’ll need to support SSL. First, kill nginx:
# service nginx stop
Now proceed through the LetsEncrypt process:
Run letsencrypt certonly
Enter your email address
Read and acknowledge the terms
Enter the domain name you chose
If the domain name has propagated (which is why it’s important to do this early), LetsEncrypt will find your server and issue the certificate in one go. If this step fails, you may need to wait a while longer for your domain to propagate so that LetsEncrypt can see it.
5. Configure nginx
With the SSL cert done, time to configure nginx!
# cd /etc/nginx/sites-available
# nano mastodon
Simply substitute your domain name where it says example.com in this snippet (lines 9, 15, 23, 24), then paste the entire thing into the file and save it.
Open up a browser tab on your domain. Mastodon can take up to 30 seconds to warm up, so if you see an error page, don’t fret. Only fret if it’s there for longer than a minute – that requires troubleshooting, which is outside the scope of this document.
You should eventually get a signup page. Congratulations! Register an account for yourself, receive the confirmation email, and activate it. This should enable you (the first user) as an administrator.
14. Securing Mastodon
This is by no means a comprehensive guide to server security, but there are two quick things you can change while the root shell is open. Start by editing the passwd file:
# nano /etc/passwd
Find the mastodon entry (it’ll be near the bottom) and replace /bin/bash with /usr/sbin/nologin. Save and quit. This will prevent anyone from logging in as the mastodon user.
Next, configure ufw. First check if it’s disabled:
# ufw status
It should be off, since this is a brand new VM. Configure it to allow SSH (port 22) and HTTPS (port 443), then turn it on:
# ufw allow 22
# ufw allow 443
# ufw enable
That will prevent any connection attempts on other ports.
If you enjoyed this guide, I’d appreciate a follow! You can find me by searching firstname.lastname@example.org in your Mastodon web UI. Give me a shout if you were able to get an instance set up with these instructions, or if you ran into any problems.
16. Upgrade to v1.2 (17 April 2017)
If you’ve installed Mastodon according to these instructions, you’ll need to do a few things to upgrade to the latest version.
Start by logging into your instance as the root user, then re-enabling your mastodon user shell (in step 14, change the mastodon user’s shell back to /bin/bash). We’ll use it in a bit to perform the upgrades themselves.
When that’s done, stop the Mastodon services like so:
# systemctl stop mastodon-*
That will shut down all the Mastodon services. In a new window, log into your mastodon user and install Ruby 2.4.1, the new preferred version:
$ cd live
$ rbenv install 2.4.1
$ gem install bundler --no-ri --no-rdoc
This will install the latest Ruby, and the version-appropriate bundler. Now pull down the latest source code:
$ git pull
There are a couple of one-time commands to run – in order, they are to install new dependencies, run database migrations, do a one-time avatar migration, and recompile the frontend.
I use Cloud9 as my primary IDE – it handles all my PHP projects, my Domo app projects, and when I start seriously developing Ionic-based mobile apps, I’ll find a way to use Cloud9 for that too.
I’ve found it particularly handy for Laravel development: A lot of the prerequisite software provided by Homestead comes pre-installed on the Ubuntu VM that Cloud9 provides, and with a few small config tweaks, it’s possible to get up and running very quickly.
1. VM Setup
One of my cardinal rules: All your code should exist in a repository. Especially with web development, there’s basically zero excuse to keeping substantial amounts of code locally.
My first step is to always create a blank repository, and use that as the clone URL for a new Cloud9 VM.
The Blank Ubuntu VM is good enough for our purposes – there’s no benefit to selecting the PHP VM, since we’ll be installing the latest PHP in any case.
2. Services Setup
Chances are, whatever you’re building in Laravel will need the MySQL service to be running, and in most cases, you’ll probably want Redis too. Cloud9 has made it a little bit more difficult to automate service startup (presumably to minimize unnecessary resource utilization), but there’s a way around that.
If you’re using a premium workspace, Cloud9 will keep it “hot” between sessions – everything you start will continue to run even after closing the IDE window itself. But that will expire after a while, and it can be a hassle to keep restarting your services.
So, the first change we’ll make to our new VM is editing the /.profile file, and adding two new lines to the bottom:
$ cd ~/
$ echo 'sudo service mysql start > /dev/null 2>&1' >> .profile
$ echo 'sudo service redis-server start > /dev/null 2>&1' >> .profile
That will start the services (and skip startup if they’re already running), won’t show any console output, and will run whenever a new terminal window is created. So now, every time you open up a new terminal window, you’re guaranteed to have those services running:
It’s not as ideal as having the server start it automatically, but in my opinion, this is a small trade-off to make.
3. PHP 7 Setup
If you’re deploying code to a new server on DO, or AWS, or Rackspace, or most other places, chances are it’s running PHP 7. By default, Cloud9 does not include PHP 7 (for reasons which escape me), so we need to take a quick detour into apt-get land.
There are two ways to set up PHP here – apt, or phpbrew. For a while I was using phpbrew, which builds PHP from source. It’s a decent option, but using ondrej/php is faster.
$ sudo add-apt-repository ppa:ondrej/php
Then you’ll need to sudo apt-get update. When that’s done, you should have access to the php7 packages:
I’m using php 7.1 here, because that’s what my DigitalOcean VM is running. Laravel will need that, plus a few other modules:
That shouldn’t take more than a minute. And now we’re good to go:
4. Laravel Project Setup
There are quite a few ways to get Laravel installed. It has a CLI-type thing that lets you do laravel new, you can clone it directly from github, or you can use Composer.
I tend to favor composer since it’s a once-off command for this workspace. There’s no benefit to installing any more tools here, since we’re only going to create 1 Laravel project in this VM. And since we’ve already got our workspace folder linked to our project repository, git can get messy.
So my favorite – use composer to set up Laravel in a temporary directory, then move it all across:
Why two move commands? The first catches all the files, but Laravel does include a few dotfiles that aren’t caught by the first move command. Sure, you could configure dotglob to modify mv’s behavior, but we’re only doing this move once for the workspace.
Just one more setup step – the database. Start by adjusting the .env file to set the username to root, and a blank password:
You can now run the initial migrations if you want.
5. Node Setup
If you’re planning on using Elixir to compile frontend assets, you’ll probably want to upgrade to the latest version of Node. Cloud9 makes this easy:
$ nvm install 7.2
$ nvm alias default 7.2
All node commands will now use the 7.2 binary. You can now install the necessary components in your workspace folder:
$ npm install
6. Running Laravel
Cloud9 features a reverse proxy – anything that binds to port 8080 in the VM will be accessible via a URL unique to your workspace. It also has a dedicated “runner” feature, which we’ll configure like so:
The name can be anything, but the run command will be:
php artisan serve --host=0.0.0.0 --port=8080
Cloud9 does make mention of $HOST and $PORT variables, but they’re always the same values anyway. Fill that in and click the Run button on the left:
Cloud9 should open up an in-IDE browser window that points to your site. You can click the little pop-out button on the right of the “address bar” to bring it up in a new browser tab.
That domain (c9users.io) will check for cloud9 authentication before proxying to your VM – in short, that URL will only work for you, and anyone else on cloud9 that you’ve shared the workspace with. Those settings are managed via the Share button on the top right:
From that popup, you can invite new users to the workspace (cloud9 does issue free accounts), or you can just make the application URL public:
You can now share that URL to anyone.
7. Get coding!
You’ve now got just about everything you need for proper Laravel development! It’s usually at this point I then take the extra step of configuring Amazon S3 as a cloud disk, push the blank Laravel project to bitbucket, and deploy my repo as a site on Forge.
This post is part of a series of posts I’m doing about Domo (the Business Cloud) and how I’m using the free tier to analyze metrics around write500.net. Click here for the full listing.
There’s a surprising amount of things to keep an eye on when it comes to a project like Write500.
Right now, only the free tier of Write500 exists – users get sent to a subscribe page, they opt-in, and receive daily writing prompts over email. It won’t always be so simple: I’ve got plans for a Plus (paid) version, and mobile apps to extend Write500’s reach.
So even this simple, free version, in terms of the architecture, already looks like this:
Every box in there has something that needs to be monitored in some way. Some of the monitoring is historical (looking back at performance over time), some of it is by exception (being alerted when something goes wrong).
That’s already a lot of potential metrics, from a lot of different sources. Individually, they aren’t particularly hard to get – it’s deciding which ones to get, how often, and what you’ll do with the information, that matters.
And that’s the product side of the equation. Not only am I building the product, I’m the one-person marketing team behind it.
Each of those boxes, too, can have a whole universe of metrics in them. Social in particular, which has a penchant for inventing crazy new metrics every few months.
Where to start?
If it seems overwhelming – it is. There’s a lot of information out there, not all of it useful. It’s way too easy to get hung up on the wrong metrics, and miss out on the ones that matter.
So the best place to start, in my experience, is notwith the data. I’ve seen too many BI projects start out with what the team could get from their databases, and in the end, they delivered dashboards that looked identical to what they were already getting from their data in the first place.
What we’re after here is actionable, meaningful insight. So without further ado:
1. The Question
A much better place to start, is the question.
If I could answer one question about your business, what would it be?
For Write500 I have a lot of questions, but in this post we’ll deal with the one that’s most important to me:
Are my subscribers getting value from Write500?
I built Write500 because I wanted to be nudged to write 500 words per day. So far, as the user of my own product, I’ve managed to achieve that – but my experience doesn’t matter nearly as much as the experience of my subscribers. If they’re not getting the value they need, then I’m either aiming at the wrong audience, or there’s a problem with my product.
Now, that question could be answered anecdotally. I could point to a few places I’ve seen bloggers start their challenge, and take that as a sign that users are getting value. I’ve seen a lot of businesses do this – make gut judgements based on limited information.
I have learned, however, not to trust my gut. Let’s quantify this instead.
Getting metrics about my subscribers is easy – I can pull the subscribers table and chart the growth. Getting metrics about the value is a little bit harder – right now, I’m not equipped for that.
Ideally, the value I want subscribers to get out of my product is producing 500 words per day – and the only way to track that metric is to have them plug their 500 words into a system where I can measure their wordcount. That’s a feature planned for the next version of Write500, but right now, I don’t have that data.
The best I can do, with the information I have, is determine whether or not users are opening the emails I’m sending on a regular basis. That’s not the best metric for whether or not they’re getting value from Write500, but I’ll operate on the assumption that consuming the prompts is at least partially valuable.
Here, already, one of the biggest problems in BI becomes visible: The gap between what you need to know about your business, and the data you have on hand to answer it. So, so many BI vendors and consultants sell dashboards that chart what you have, but don’t give a second thought to what you need.
2. The Data
So based on my question, what data can I acquire to answer it?
Write500 has a bespoke tracking pixel. Every prompt that gets sent is uniquely identifiable, and includes embedded tracking that fires (email client willing) when the email is opened. This data is sent straight back to my server, instead of going through a third-party analytics tool.
The raw data, as you can see in the screenshot, includes the GUID for each send, and the server-side timestamp for when the email was opened. I also have the timestamp for when the email was sent. With this data, I can answer two sub-questions:
Are subscribers opening the emails they’re receiving?
Do subscribers open the email on the same day they receive it?
Together, the answers to those questions would at least indicate whether or not the emails themselves are being utilized (and whether my subscribers are getting value). So looking at the data I have available, this is what I can get to help answer those questions:
Per subscriber: (dimension)
The amount of prompts they have been sent (metric)
The amount of prompts they have opened (metric)
The open rate (calculated)
Per day: (dimension)
The total amount of prompts sent (metric)
The amount opened within 24 hours (metric)
The amount opened within 48 hours (metric)
The amount opened within 72 hours (metric)
The amount opened after 72 hours (metric)
The amount not opened (metric)
Dimensions and Metrics
I should explain what these are before continuing, and I’m doing this in the context of Domo’s capabilities. Within any tabular data set, you’ll have two types of fields:
Everything that’s not numbers
Metrics are numbers you can measure, and Dimensions are the categories you group/slice/breakdown those metrics in. So:
New Subscribers per day
Recurring Visitors by mobile device
Purchases over $50 by persona
In the samples above, Day is a dimension (because it’s not a number). But Subscriber is also a dimension, even though it’s technically a numeric ID – except that it’s a number that makes no sense to measure. This is the exception to the rule.
It can’t be summed, averaged, min’d or max’d and present any meaningful result: What’s the point of knowing the average subscriber ID in your database? The count of subscribers is more useful, and in that case, the count becomes the metric.
3. The Acquisition
So now we know what question we’re answering, and the data that we’re going to use to answer it. This is where we start getting into the implementation.
The first step, obviously, is to acquire the data. In a previous post, I dealt with this issue for Write500, eventually settling on a SFTP upload for my subscribers dataset. Since then, I’ve added a better stats system to the app.
So for Write500 specifically, I now have this happening:
I have a series of Artisan commands, each of which generates a single CSV file. Those commands are scheduled using the cron system on Forge, and when they run, the results are uploaded to a dedicated S3 bucket. Domo, in turn, pulls that information into its Vault:
Now that I have the data coming in, it’s time to build some charts!
Visualization is a problem domain all its own, and has way too much science behind it to cram into a 3000-word blog post. However, we’re doing visualization for BI, which is very different to visualization for analysts, engineers and scientists.
BI visualization has to meet several fundamental criteria (at least, the type of BI I do):
Self-contained – the chart should explain what it is without requiring an appendix. Any chart that has a “How to read this chart” next to it, is a bad chart.
Key call-out – Whatever the purpose of the chart is, it should be obvious at a glance, preferably from across the room.
Honest – Too many “analysts” will fiddle with colors, min/max axes dimensions, sorting and scale to try to massage the truth behind the numbers. Don’t.
Actionable Next Step – In charts that are meant to drive decision-making, make the decision obvious.
In my experience, this tends to mean the following:
You’re going to re-use a lot of the same 3-4 chart types (line, bar, area, donut)
You’re going to create lots of smaller charts, instead of consolidated/complex ones
Colors are really important
Short titles are really important
I will eventually build the charts that answer my key question, but let’s start with some simple ones.
Domo includes the ability to group charts into Collections, which make it simpler to manage complicated dashboards. In this case, all the charts that pertain to my subscribers will go in one collection – other collections will include marketing and product data.
Note: All the charts here are built on real data, as of today.
0. Warming up
Easiest thing to chart – one metric vs a date.
Here’s an example of what I think is a good chart:
The title and subtitle make it clear what I’m looking at,
The summary number makes the current status clear,
The bar chart shows the growth nicely, and
Using Domo’s linear regression Projection feature, I can see that I should end today with over 100 new subscribers
So overall – I’m getting new subscribers, and it’s visually obvious from the chart that the velocity is increasing.
But now let’s answer our first sub-question:
Are subscribers opening the emails they’re receiving?
Now we get to define a proprietary metric! I’m going to create a metric called “Engaged”:
Engaged: A user that has no more than 1 unopened prompt email
The logic behind this: Users receive one email per day, and there will be a gap between receiving an email and opening it. However, for as long as that gap exists, the email will show as “1 unopened” in my data, which is not fair – the user just hasn’t had a chance to open the most recent email yet.
“2 unopened” means they’ve neglected at least one email, so I’ll consider them less engaged. Yes, this is a high standard.
How do we calculate this metric? Here’s the Beast Mode formula I’m using, directly in the card.
This gets us the total users that have between 0 and 1 unopened emails, as a proportion of the total number of users in the data.
That can be represented in a single large number card, like so:
That’s a pretty impressive result! Almost 70% of my subscribers are opening their emails on a regular basis, and it could be even higher: remember that I cannot track opens from email clients that block the call to my tracking pixel.
I also want to get a sense of the proportions, though – so we’ll take that summary metric and use it as a summary number on a pie chart. I’ll group my subscribers like so, using another Beast Mode.
Honestly, these sorts of CASE statements represent 90% of what I use Beast Mode for, in Domo – to create custom categories.
We’ll apply that formula to the data, and spin up a pie chart. Here’s what we get:
So of all my subscribers, 60 of them are neglecting their prompts completely. This is also useful information – I know who those subscribers are, so I can segment them out and try different email subject lines, to see if I can maybe get them more engaged and bolster the overall rate.
There’s one more chart we can create – we can compare our engagement rate to an industry open rate. Using this source data, I’ll take Hobbies as my benchmark.
Not bad at all! But then, that’s to be expected – my emails are not marketing, they’re more transactional in nature, and transactional emails have much higher open rates.
So let’s go back to our question:
Are subscribers opening the emails they’re receiving?
Resoundingly, yes: Close to 70% of my subscribers are opening their emails.
On to the next question!
Do subscribers open the email on the same day they receive it?
The answer to our previous question can cover this one too – if Unopened is between zero and one, then yes – most open the same day they receive it. However, that’s an aggregate view, and I want to see some more specific on how long it takes subscribers to engage with their emails.
So for this, we’re looking at the time difference between the email being sent, and opened. I’ve already done the heavy lifting of categorizing the data inside my data-providing Artisan commands, so the data I’m visualizing is already close to useful:
Our last charts looked at Subscriber Engagement, which is a higher-level metric than the actual individual emails being opened or not. For these charts, we’re taking a different angle, looking at the actual emails. We’ll get similar rates, but not identical – that’s to be expected.
Of all the emails that do get opened, practically all of them get opened on the same day (within 24 hours of being sent). And it looks like, overall, about 40% of the emails sent by the system have not been opened yet. Or, at least, they haven’t fired the tracking pixel yet.
The thing about tracking email open rates in particular, though – naturally, it’s not a trendable metric.
For example: You can send 100 emails on Monday, then calculate the open rate every day of the week, and get a different (higher) number every day.
Why? Because subscribers take time to open their emails (if at all). So looking at that chart above, right now, 40% of the emails I’ve sent are unopened. Those emails could have been sent today, or a week ago.
Right now, I care about whether or not subscribers open on the same day – and according to this chart, nearly all opened emails are opened on the same day. Emails opened at +1 Day and up are completely dwarfed. This is what happens when we remove Unopened emails from the chart:
So that’s my question answered!
5. Taking Action
I’ve got my questions, they’re being answered by data, and now I need to think about the next step in this: What actions can I take, based on this data, to drive my business goals?
Right now, my goals are two-fold:
Drive more subscribers to Write500
Improve the email open rate
For (1), what I really need to look at is my subscriber acquisition machine – the marketing I’m doing, which is outside the scope of this post. I do plan on writing another one dealing specifically with marketing metrics though.
For (2), I now have a tight group of subscribers that are either not opening their emails, or not opening them on a daily basis – meaning they’re not writing every day. I can segment these users out and try a few things to remedy that.
I could also simply reach out to the subscribers that haven’t opened any emails yet. There might be a good reason why those emails aren’t being opened:
maybe they’re getting caught in spam,
maybe the user hasn’t been at their mailbox in the last week, or
maybe they no longer want to receive them, but haven’t bothered opting out.
My action would be to find out what it is, and remedy it where possible.
6. (Finally) Measuring Results
The final piece of the puzzle, and the one question that every BI dashboard should answer:
Are our actions having the intended effect?
Since the insights and actions are data-driven, the results should be too. For every action that you take to impact your metrics, you need to think about how you’re going to measure the results in a quantifiable way.
Let’s take the Open Rate chart above. When I start taking action to improve it, I expect to see that, over time, the overall percentage of Unopened emails goes down.
Since it’s an email open rate though, I can only get snapshots at regular intervals. The data generator I wrote does just that – it gathers point-in-time statistics, and appends them to the dataset. Right now, it’s running once per day, and it’s grouping the sends by the date they went out.
So putting that into a 100% Stacked Bar Chart, I get this:
Every day, I will get another bar on this chart, and the chart itself will be completely recalculated (if emails sent on previous days are opened, which we know from the data above it will be).
However, over time, that red portion should get smaller. If I wanted to create very precise tracking on this, I’d create a recursive dataflow to record the Unopened rate in a dedicated dataset – but for right now, I think I have enough BI going on!
I hope this post has been informative!
I’m planning on doing a deeper dive post into Write500’s metrics at the end of the month – that post will deal more with my expectations vs the reality, the charts I ended up using on a daily basis, what decisions I took, and how I measured those results.
If you want to be notified when I write more of this stuff, check the Subscribe widget in my sidebar, top right. Or you can follow me on Twitter.