It’s Good to be Home (#7 of 52)

This post is more than a year old. The information, claims or views in this post may be out of date.

A Saturday post on an actual Saturday – madness!

Last week was hectic – pretty much a repeat of the week before. The second half of my first-ever full-time on-site consulting gig drew to a close, but not before leaving me with a task list a mile long. It turns out that delegation really is an important part of building a business – including delegation to my future self (or I’d go insane).

At least I’m home now! Slept in my own bed last night, which was glorious. Overslept, really.

So it should come as no surprise that my habit tracking went to shit again over the last week. It’s pretty obvious to me now that maintaining these habits does require a bunch of external inputs (namely, being home every morning), and it further diminishes the notion that I’ll take up digital vagrancy at any point in the future.

At least I recovered from that gastro bug, which was no fun at all.

The last week also presented me with an opportunity that I’m now seriously considering: Taking on a full-time role at the company I consulted, to help them fix their IT and systems, helping them get to the next level.

In many ways it’s my dream job – it won’t last forever, I’ll have a meaningful remit, the authority to implement the changes that make sense, and I’ll walk away from it with quite a lot to show. For the first time, I’ve got tangible validation that my skillset is, in fact, pretty rare – and I’ve now got an opportunity to really use it.

It’s also causing a massive internal debate, once again ripping up the floorboards of all my assumptions.

On the one hand, as a career move it makes total sense. Pushing out and establishing myself as an independent consultant here will pay huge dividends well into the future. The job itself is challenging, but not impossible. The tradeoffs may well be manageable, too, and it’s likely to open up new career paths and opportunities in the future.

But on the other hand: Do I really want to do this?

A big part of leaving full-time employment was specifically to push myself into new areas of personal growth. All things considered, this job wouldn’t really be that much of a push – it’s something I’m good at, and can do at length, but it won’t be as personally challenging as a total career change.

Doing this well (and establishing those new opportunities for myself down the road) will only serve to further draw me away from the reasons I quit this world in the first place. Five years hence, I’ll find myself doing the same things I was doing five years ago, and again I’ll feel disappointed at my lack of taking actual risks.

I recognize that I’m really lucky to even be in this position: Openly debating whether or not to take a potentially career-defining job, over doing something completely unrelated for a career. But I’m also the one that has to wake up and do this job every single day, and at some point I also have to consider what I want for myself.

Nothing’s been decided yet, other than the fact that I’m thinking through all of this very carefully. In the end, I doubt I’ll walk away from the opportunity – it’s more likely that I’ll deliberately carve out the time for the things I really want to try.

Everything else in the last week has really just been implementation details. I’ve once again found myself hand-rolling a 2-tier analytics and reporting stack, purely because rolling out any of the free tools would have added far more complication too early in the process:

Most of my week in one simple diagram

Question: How do you link records between two systems when no correlation or shared key already exists? Answer: You create a system that goes hunting across two databases, following ever-broadening match rules in an attempt to best-guess what those correlations should be. A solid week of my life for the first version of that.

I’ve also regained an appreciation for communication as a skill. It’s actually pretty important that people talk to each-other, and not past each-other. The latter is how you end up with organizations that make absolutely no sense.

Finally, I’m continually vindicated of my decision to leave full-time employment. We’re about 10 weeks away from my 1-year anniversary, and a part of me was worried that I’d come to regret the decision to leave. So far, no such luck.

Other than that, not too much to report. I’m doing badly on my reading goals – Best Dick is still on my reading list, and I haven’t touched it for at least two weeks. I’m at least keeping up with 1 post per week on blogging, even though the goal was 3 per.

Mostly I’m still trying to manage an escalating-workload situation. A big part of my paranoia thus far has been the upcoming CIT payment, and to what extent that’s going to eat into my cash buffer. At one point I was within striking distance of having a 6-month buffer built up, but that’s about to be wrecked. I just hope the damage isn’t too bad.

I’m also dealing with the growth question – and if I want to do any of that at all. It goes against everything in my nature to see a problem I can solve, and then walk away from it – but I’ll have to start doing that in order to maintain my sanity. I’m confident I could spend the rest of my life fixing things and getting rewarded for it, while never managing to pursue the things I’ve wanted to do since I was a child.

Or maybe now is the right time to accept that not all dreams come true, and go with what’s actually around me.

Like I said – hectic week.

Also:

New episode of The Noscript Show came out today. We missed a lot of international tech topics in the pursuit of dealing with local issues, the peak one being loadshedding – and how we absolutely shouldn’t have had to deal with it.

The Helderberg Dev Meetups were postponed last week (bad scheduling) but are back on as of the 19th. I’m still not sure whether or not I’m preparing anything for it, but I’m looking forward to a relaxed evening of chilled conversation.

In the coming week it’s back to something resembling business-as-usual, as I attempt to negotiate an exit path for a project that’s dragged on far too long, and start onboarding my second (!) part-time freelancer. I guess this is how we all learn!

Next year’s growth market: Smart Speaker development

This post is more than a year old. The information, claims or views in this post may be out of date.

It’s officially the last Monday of 2018 – wild! We made it to the end of the year and everything is still more or less intact, who would have guessed?

If you’re looking for a good New Year’s Resolution for 2019, might I suggest learning how to build integrations for smart home speakers?

The smart speaker market reached critical mass in 2018, with around 41 percent of U.S. consumers now owning a voice-activated speaker, up from 21.5 percent in 2017.

Sarah Perez, Techcrunch (link)

What this means in practice is that 41% of American consumers are now on a new platform with big commercial potential, and chances are it’ll continue to grow over the next few years.

In effect, this is a whole new market – the next iteration of the mobile platform/app wars, except there’s a lot less UI overhead.

It also forecast that Alexa would generate $18 billion to $19 billion in total revenue by 2021 — or ~5 percent of Amazon’s revenue — through a combination of device sales, incremental voice shopping sales and other platform revenues.

Sarah Perez, Techcrunch (link)

The magic ultimately comes down to the integrations. Alexa calls them “Skills”, Google calls them “Actions”, but they’re the same thing: Writing conversational dialog that lets end-users engage with your service verbally.

This was the market that Siri was supposed to unlock, but it turns out that people already on their phones are more likely to use their phones to get things done, instead of switching over to a clunky voice interface.

The most simple explanation is that app developers have limited resources, and don’t see the point in supporting Siri when users aren’t demanding it.

Jared Newman, Fast Company (link)

Home speakers are a different story. They’re always on, don’t require you to physically interact with them, and have the effect of enabling any given room with something approaching ambient intelligence – all with purpose-built microphone arrays and optimizations for working in home environments. Basically, for quick commands, they’re going to be more useful to more people.

It already looks like Alexa has taken the lead on revenue potential (it’s backed by a global retail giant, so this is not surprising), but both Alexa and Google Home would be good targets to build for.

Side note: Apple’s HomePod devices are nowhere to be found (5% marketshare), and their SDKs and learning requirements tend to be a lot steeper than Amazon or Google, so if you’re not already in that ecosystem it might not be worth your time to start.

Google’s platform is called Actions, and you can learn more here: https://developers.google.com/actions/

Alexa’s platform is called Skills, their developer site is here: https://developer.amazon.com/alexa

If you’re new to the world of building conversational interfaces, I strongly suggest starting with the simpler apps:

Google’s Dialogflow: https://dialogflow.com/
Voiceflow for Alexa: https://www.getvoiceflow.com/

This is all within the same problem domain as chatbots (you’re creating conversational flows with voice instead of text), so anything you learn here is going to be easily transferable to other platforms and problems. Well worth your time, in my opinion!

For myself though, I doubt I’ll ever own a smart speaker at home (the creep factor is a bit too much for me), but I might end up building one or two integrations for any products I end up developing.

Are you going to try developing something in this area next year?

Integrating Laravel and Flarum

This post is more than a year old. The information, claims or views in this post may be out of date.

One of the features I wanted to add to Write500 was a community forum. I’ve worked with Flarum before, and while it’s still ostensibly beta software, it has some really nifty features and works quite well across desktop and mobile. I decided early on that I’d plan to integrate a standalone instance of Flarum into Write500.

Flarum has a JSON API, which as it turns out is really easy to use (despite being insufficiently documented at the moment).

I wanted to set up an integration such that Write500 handled the authentication (you could only log in from the site, not the forum itself), and that changes to a user’s Write500 profile would be synced across.

Laravel Flarum - Page 1 (4).png

This integration broke down into several parts. I’ll go into detail on each one here – hopefully it’s useful!

Contents

  1. Laravel Setup
  2. Flarum Setup
  3. Integration: Creating a new user
  4. Integration: Authenticate the user
  5. Integration: Avatars
  6. Conclusion and Additional Notes

Laravel Setup

At the outset, I implemented a “nickname” field on my User model, with the following constraints (in the validator() method in RegisterController).

'nickname' => 'required|max:32|unique:users'

Those are the same constraints Flarum uses. So now, when I integrate the two (using the nickname as the Flarum username), I can be at least 99% sure it’ll work every time – barring the inevitable weird-unicode-character issue, or some other edge case.

I also ended up adding the following to my User table:

$table->string('nickname');
$table->integer('forum_id')->nullable()->default(null);
$table->string('forum_password')->nullable()->default(null);

Those fields will be populated when the integration runs, and are critical for everything else.

Finally, I needed a few configurable fields. I added these to my config/app.php file:

'forum_url' => env('FORUM_URL', 'https://forum.write500.net'),
'forum_admin_username' => env('FORUM_USER', 'default_admin'),
'forum_admin_password' => env('FORUM_PASS', 'default_pass'),

Note that in the samples below, I don’t reference any of these fields directly.

Flarum Setup

There are a few tweaks made to Flarum to enable all of this. In summary:

  • Installed the latest version of Flarum
  • Set up HTTPS using LetsEncrypt
  • Added the Flarum Links extension
  • Set up the default admin user as a generic account (not named after myself)
  • Disabled new-user registration

About the Links extension

When a user navigates to the community, I want them to be able to navigate back to any other part of the site, without feeling like they’ve really left Write500. To do this, I used the Links extension to create a navbar with the same links as on the main site.

Screen Shot 2017-02-13 at 2.31.49 PM.png
Header on main site
Screen Shot 2017-02-13 at 2.31.58 PM.png
Header on Flarum

I could have used Flarum’s Custom HTML Header feature for that, and created my own menu bar to replace the forum’s one, but that would have lost me the search bar and notification indicator.

About the default admin user

I use the username and password for the admin user to drive the integration. I could technically create an API key on Flarum’s side and just use that instead (it would save me a CURL call), but that also means going directly into the Flarum database to do stuff, which I wasn’t keen on.

One day, when you can generate API keys in the Flarum interface itself, that would obviously be the way to go.

Note about the code samples

The code samples below (all the gist embeds) are not taken directly from my Write500 source – they’ve been adapted for standalone illustration.

Integration: Creating a new User

My first integration requirement:

Whenever a new user is created in Laravel, that same user must be created and activated in Flarum.

The following has to happen:

  1. Authenticate to the Flarum API with my admin account
  2. Create a new user
  3. Activate that user
  4. Link that user to our Laravel user

I ended up putting all of that in a Job, and dispatching it when a new User object was created (in my RegisterController):

dispatch(new \App\Jobs\SyncUserToForum($user));

And now for the actual code!

Step 1: Authenticate

<?php
// Details to access Flarum
$api_url = "https://my.forum.url/api/token&quot;;
$username = "my_flarum_user";
$password = "my_flarum_pass";
// CURL call
$ch = curl_init($api_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Content-Type: application/json'
]);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
'identification' => $username,
'password' => $password
]));
$result = curl_exec($ch);
$session = json_decode($result);
$session->token; // API Token
$session->userId; // Authenticated user ID
view raw auth.php hosted with ❤ by GitHub

Nothing complicated here – just POSTing to the API to get a token back for this session. This would be the step I could skip if I had a preconfigured API token.

This same method is used whenever I need to log into the Flarum API, so for brevity I’m excluding this code from the rest of the samples.

Step 2: Create a new user

<?php
$api_url = "https://my.flarum.url/api/users&quot;;
$token = $session->token; // See: https://gist.github.com/woganmay/88f15e96fc019657a0e594366403b5cf
// This must be a token for a user with Administrator access
$new_username = "johnsmith";
$new_password = "password1234";
$new_email = "john.smith@example.org";
$ch = curl_init($api_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Content-Type: application/json',
'Authorization: Token ' . $token
]);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
'data' => [
'attributes' => [
"username" => $new_username,
"password" => $new_password,
"email" => $new_email
]
]
]));
$result = curl_exec($ch);
$new_user = json_decode($result);
$new_user; // Will be a large JSON object containing the new user's details:
/* SAMPLE
{
"data":
"type": "users",
"id": "1",
"attributes": {
"username": "johnsmith",
"joinTime": "2017-02-11T16:34:40+00:00",
"isActivated": false,
"email": "john.smith@example.org",
}
}
*/
view raw new-flarum-user.php hosted with ❤ by GitHub

I should, strictly, be checking the API first to see if that nickname or email exists – to prevent any attempts to create duplicate users. However, both Laravel and Flarum itself should fail a new registration if the nickname or email address fields are not unique – so I should be okay there.

Step 3: Activate the user

<?php
$api_url = "https://my.flarum.url/api/users/&quot; . $userid; // See: https://gist.github.com/woganmay/4c15a0f7c16e41ab3a3ea1a73c595bf9
$token = $session->token; // See: https://gist.github.com/woganmay/88f15e96fc019657a0e594366403b5cf
// This must be a token for a user with Administrator access
$ch = curl_init($api_url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "PATCH");
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Content-Type: application/json',
'Authorization: Token ' . $token
]);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode([
'data' => [
'attributes' => [
"isActivated" => true
]
]
]));
$result = curl_exec($ch);
$user = json_decode($result);
$user; // Will be a large JSON object containing the user's details:
/* SAMPLE
{
"data":
"type": "users",
"id": "1",
"attributes": {
"username": "johnsmith",
"joinTime": "2017-02-11T16:34:40+00:00",
"isActivated": true,
"email": "john.smith@example.org",
}
}
*/

This could possibly be done in the registration call itself (sending isActivated=true as an attribute upfront), but I’m breaking this out for another reason.

I know that, at some point, I’m going to set up a registration workflow on Write500 that requires users to activate their account before they can log in, and I’m only going to trigger the Flarum activation at that point.

What this means, practically, is that a user’s nickname will be reserved across the system, but will only be @-mentionable once their Write500 account is activated.

Step 4: Link to our Laravel user

When I created this new user, I generated a random password like so, which was used to create the user on Flarum:

$generated_password = substr(md5(microtime()), 0, 12);

The user never has to know this password, since the actual authentication is handled by the site. So we just need to update a few things on our user model:

// $new_user is the result of the POST /api/users call
$user->forum_id = $new_user->data->id;
$user->forum_password = $generated_password;
$user->save();

The forum user ID is not strictly required, but it makes it easier to check whether the user has a forum account yet or not – we don’t want to present the sign-in link to the user before they have a forum account to sign in to:

if($user->forum_id != null) { /* Integrated successfully */ }

Summary

So we now have a job that registers a new user account on Flarum once we have a valid Laravel user – none of it requiring any modifications to Flarum’s internal database.

Integration: Authenticate the user

Next, we need users to be able to authenticate to the forum.

Create a one-click link on Laravel’s side that authenticates and redirects the user to the Flarum instance.

There are a couple of ways to handle this, and I’ll admit upfront that the solution I’ve come up with (as of right now) may not be the best, security-wise.

The following has to happen:

  1. Create a new session token
  2. Set that as a cookie, then redirect to the forum
  3. Update the forum to only allow authenticated users access

Flarum makes this quite easy – the Token you get from the /api/token endpoint can be used as a flarum_remember cookie.

Step 1: Create session token

Once a user has been provisioned in Flarum (they have a numeric forum_id value on their model), we can use their nickname and password to generate a new session token, in the same way we authenticated the admin user.

Step 2: Set token value as cookie

I butted heads with this step for a while, before coming to a solution. On the Flarum side, I had to create a new auth.php file in the forum web root (where index.php and api.php are), with the following:

All that file does is take whatever token value you give it, set it as a cookie, and then bounce you to the forum homepage.

In terms of security, this setup is making me a bit nervous. However, it’s worth noting:

  • The token is sent over HTTPS, and HTTPS URLs and parameters are encrypted
  • The only way to hijack someone else’s account using this, would be knowing their session ID – at which point you can manually add that as a browser cookie anyway
  • The only other exploit that’s possible here is someone trying a script injection attack, but again, that’s nothing you cannot set in your browser manually
  • If a user tries hitting this route manually with a bad token value, they’ll be bounced to the forum, invalidated (see below) and then sent back to the login page.

With that script set up, you can now redirect the user to it from Laravel (as in Step 1), and they’ll be bounced to the forum, fully authenticated.

return redirect()->to('https://my.flarum.url/auth.php?token=' . $session->token);

Step 3: Only permit authenticated users to view the forum

This was another ugly script hack. Flarum itself lets you set permissions so that guests cannot read content – they’ll just see a “no discussions” message when opening the forum.

However, since I’m also planning on removing the Login button on the forum, the only way an unauthenticated user will be able to log into the forum, is by going via my login page.

So I need to add a redirect before Flarum itself fires, to check whether the user is authenticated. I ended up with this:

<?php
if (isset($_COOKIE['flarum_remember']))
{
$mysql = new mysqli("localhost", "flarum_sql_user", "flarum_sql_pass", "flarum_database");
$result = $mysql->query("SELECT * FROM access_tokens WHERE id = '". $mysql->real_escape_string($_COOKIE['flarum_remember']) ."'");
if ($result->num_rows == 0)
{
// Invalid cookie
header('Location:https://my.laravel.site/login&#39;);
exit;
}
}
else
{
// No cookie
header('Location:https://my.laravel.site/login&#39;);
exit;
}
// Rest of normal index.php here
view raw index.php hosted with ❤ by GitHub

When a user authenticates with Flarum (which all runs via the api route eventually), a new access token is generated and stored in that table. If the value matches, they’re logged in – if not, they’re bounced to my login page. After logging into Write500, hitting that Community login route will authenticate and take them through.

That could be improved by reading the database connection details from config.php instead, but those values are not likely to change for this setup. I’d also have preferred it if Flarum used some sort of file storage for sessions, in the same way basic Laravel does – but we can’t always get what we want 😉

Integration: Flarum Modification

So we now have the basics down – new users are registered, users can log in via my site, and users cannot reach the forum without being authenticated.

Modify Flarum to integrate the navigation with the main site, and prevent users from logging out via Flarum.

On the Flarum side, I want to change what some of the buttons in the navigation do – for instance, they shouldn’t be able to log out via Flarum (since that still leaves them logged into Write500), and I don’t want them to be able to manage their forum settings directly in Flarum.

After some trial and error, and almost giving up and hacking on the core scripts, I eventually landed on this neat hack.

In the Flarum administration section, under Appearance, you can add your own custom HTML. I used that to deploy a script block that looks like this:

<script type="text/javascript">
var thisHackInterval = setInterval(function(){
// Check if app is available yet
if (typeof(app) == "object")
{
// Set app routes to whatever I need
// This will repaint the DOM automatically
app.routes.settings.path = "https://my.laravel.site/preferences&quot;;
// Remove navigation elements I don't want
document.querySelector("li.item-logOut").remove();
document.querySelector("li.Dropdown-separator").remove();
clearInterval(thisHackInterval);
}
}, 500);
</script>
view raw custom-header.html hosted with ❤ by GitHub

That script is loaded in the page header, and executes immediately. At the end of the page, Flarum loads its app script and creates an app object. Before that happens, there’s nothing to modify, and I cannot force my javascript to run at the end of the page. So instead, my script keeps checking to see if the app variable is available, and when it is, it makes some changes.

With that script in place, the forum is now about as integrated as it can get – there’s just one more thing to cover – Avatars.

Integration: Avatars

Users can upload and crop avatars to Write500.

Automatically synchronize the user’s avatar from Laravel, to Flarum.

I wanted those avatars to sync across to the forum, whenever they were created or updated on Write500’s side. I also have a “Reset to Default” which deletes the current avatar, and that deletion should be synced across as well.

Delete Avatars

This one was pretty easy. On the route that resets my avatar, I dispatch a job that basically does this:

<?php
// See: https://gist.github.com/woganmay/4c15a0f7c16e41ab3a3ea1a73c595bf9
$api_url = sprintf("https://my.flarum.site/api/users/%s/avatar&quot;, $user->id);
// See: https://gist.github.com/woganmay/88f15e96fc019657a0e594366403b5cf
$token = $session->token;
$ch = curl_init($api_url);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, "DELETE");
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'Content-Type: application/json',
'Authorization: Token ' . $token
]);
$result = curl_exec($ch);

It authenticates to the Write500 API, and issues a delete request against the avatar resource. Simple.

Upload avatars

This one’s a bit more complicated. The API endpoint is expecting a form-data upload, which takes a bit of fiddling to get right on PHP’s end.

After a user uploads, crops and saves an avatar, it’s stored in the public/ directory with a GUID, which is saved against the User model. All I really need to do is read that file from disk and upload it to the Flarum API. Which I achieve like so:

<?php
// See: https://gist.github.com/woganmay/4c15a0f7c16e41ab3a3ea1a73c595bf9
$api_url = sprintf("https://my.flarum.site/api/users/%s/avatar&quot;, $user->id);
// See: https://gist.github.com/woganmay/88f15e96fc019657a0e594366403b5cf
$token = $session->token;
$avatar_file = "/storage/avatars/1234.jpg"; // Absolute path preferred
$ch = curl_init($api_url);
curl_setopt($ch, CURLOPT_HTTPHEADER, [
'User-Agent: Opera/9.80 (Windows NT 6.2; Win64; x64) Presto/2.12.388 Version/12.15',
'Content-Type: multipart/form-data',
'Authorization: Token ' . $token
]);
curl_setopt($ch, CURLOPT_POST, TRUE);
curl_setopt($ch, CURLOPT_POSTFIELDS, [
'avatar' => curl_file_create($avatar_file, 'application/octet-stream', 'avatar')
]);
$result = curl_exec($ch);

That bit of code will blindly upload a new avatar. Flarum doesn’t require you to delete an existing avatar before uploading a new one, so I can trigger this all I want. Right now, it’s just dispatching as a job after a new avatar is saved in Write500.

I’m not sure whether or not the User-Agent is strictly required for this, but I haven’t had time to go back and test it. I do know that the Flarum API does behave differently depending on whether it thinks it’s being hit from a browser.

Conclusion

So that’s an exhaustive look at how I’ve enabled the following integrations:

Laravel Flarum - Page 1 (5).png

I’m sure that some elements of this could be improved, and if I make any substantial updates to this, I’ll likely write a follow-up post here.

Additional Considerations

This is all still a work in progress, and I’m sure I’ll find more bugs as I go along.

Right now, for instance, I think it might make more sense to keep the user token cached in the User model, and only issue a new one when the current one expires. Right now, if you upload an avatar (for instance), it generates a new token that might invalidate the old one. So you’d have to log into the community from scratch to get access again.

There’s also notification integration. There’s an API endpoint that returns information about new notifications (@-mentions, etc), which I would want to display in the Write500 navbar.

I’d also want to take over the profile view eventually. I’m aiming to build a user profile page in Write500 to show some high-level stats, and I’d rather integrate forum activity in there, as opposed to having Flarum display any user profile page at all. Finally, I need to consider what will happen when users cancel their Write500 subscriptions.

Those are all questions for another day though!