With the deprecation and planned decommission of both the Legacy HTTP FCM API and the Firebase Admin SDK Batch Send API (sendMulticast() method), it has become a challenge to send multicast notifications at scale and with low latency to large numbers of devices simultaneously without relying on the deprecated APIs.

Since the new FCM HTTP v1 API only accepts a single device token per request, we would essentially have to send an entire HTTP request per device token, per notification. At scale, sending to large numbers of devices (1M) would be extremely slow over HTTP 1.0 / 1.1 and require lots of TCP connections.

I was surprised to discover that the official Firebase Admin Node.js SDK had been updated to include a new method sendEachForMulticast(), and developers are urged to migrate to using it instead of the now deprecated sendMulticast(). However, if you were to use this method with any large number of device tokens, your server might crash or grind to a halt, as the Firebase Admin SDK opens a new HTTP 1.0 connection for every single device token simultaneously.

Thankfully, the new FCM HTTP v1 API also supports HTTP/2 connectivity. With HTTP/2, a single connection (referred to as a session) can be kept open and multiple requests (referred to as streams) can be simultaneously sent over this single connection. The FCM HTTP v1 API allows up to 100 simultaneous streams per session, and we are free to open multiple HTTP/2 sessions to improve concurrency and deliver the notification ASAP to as many devices as possible.

Strangely enough, the Firebase Admin Node.js SDK has not been updated to utilize HTTP/2 under the hood for its multicast functionality. So I took matters into my own hands and published a package to npm that does just that.

First, install the package using npm:

npm install fcm-v1-http2 --save  

Then, start using the package by importing and instantiating it:

const fcmV1Http2 = require('fcm-v1-http2');

// Create a new client
const client = new fcmV1Http2({  
  // Pass in your service account JSON private key file (https://console.firebase.google.com/u/0/project/_/settings/serviceaccounts/adminsdk)
  serviceAccount: require('./service-account.json'),
  // Max number of concurrent HTTP/2 sessions (connections)
  maxConcurrentConnections: 10,
  // Max number of concurrent streams (requests) per session
  maxConcurrentStreamsAllowed: 100
});

// Populate array with any number of FCM device tokens
const tokens = ['ccw_syAXSNOY9ml-Kqh9wo:APA91bHAEQccW1ZpbPvsGc0LFyjEthAt_GZO7HkBGiKounM................uIDEHijb4UR5f3dhyjhO5IbiWhJAA7RVp63KSFCg384PR7nfKADReWUONEJlCnHo15WwZagVTmFcgW'];

// Set FCM API v1 message params
// https://firebase.google.com/docs/reference/fcm/rest/v1/projects.messages#Message
const message = {  
    data: {
        // Set custom payload
        message: 'Hello World'
    },
    android: {
        // Burst through Doze mode
        priority: 'high'
    }
};

// Send the notification
client.sendMulticast(message, tokens).then((unregisteredTokens) => {  
    // Sending successful
    console.log('Message sent successfully');

    // Remove unregistered tokens from your database
    if (unregisteredTokens.length > 0) {
        console.log('Unregistered device token(s): ', unregisteredTokens.join(', '));
    }
}).catch((err) => {
    // Sending failed
    // Log error to console
    console.error('Sending failed:', err);
});

Let me know what you think, and if you faced any issues using the package!

I recently finished setting up a Lead Generation ad on Facebook Ads, which is a special type of ad that asks a potential customer to leave their details for you to get in touch with them. The process looks something like this, and Facebook pre-fills any information they already have about your customer:

So far, so good. For the advertiser, though, to actually get access to the incoming leads (and the submitted form info), you can either:

1. Connect to a CRM of choice ($$) or a tool such as Zapier (also $$)
2. Manually go into the Facebook Business Suite -> Select Your Account -> More Tools -> Instant Forms and click Download to fetch a .csv file with any new leads every time you get a new lead (by manually checking the page for new leads)
3. Subscribe to the leadgen webhook which is invoked every time a lead submits the form

Since time is of the essence with most leads, and since I chose to go with the free route, option 3 made the most sense. Yet actually getting it to work was the tricky part. Here's a detailed step-by-step guide on how to do just that:

Create a Facebook App

1. Sign up for Facebook Developer access if you haven't already, from the same account as your Facebook Ads advertising account.

2. Create an app and select Manage business integrations as the app purpose. If you're using Facebook Business Manager, make sure to select the right Business Manager account in the app creation form, when asked to do so.

Note: You will need to submit your app to Facebook for review to enable Live mode, otherwise you won't be able to access real leads but only test leads.

Generate a Never-Expiring Page Access Token

1. Visit the Graph API Explorer and add the click Generate Access Token:

   2. Add the following custom permissions which are necessary to retrieve Facebook Ads leads:

pages_show_list
ads_management
ads_read
leads_retrieval
pages_read_engagement
pages_manage_metadata
pages_manage_ads

   3. Click Generate Access Token again and select your page, granting all permissions:

   4. Under User or Page, select the name of your page, ensure the permissions are still there, and click Generate Access Token:

Accept the permission dialog once again.

   5. Most likely, the page you selected under User or Page, will have reset back to User. Select your page again, and click Generate Access Token:

   6. Now, let's extend the Access Token's expiration date. Copy the Access Token and paste it into the Access Token Debugger, and click Debug. By default, page access tokens expire in an hour, but we can extend them indefinitely.

   7. To extend the access token, scroll down and click the Extend Access Token button at the end of the page:

   8. Click Debug and ensure the expiration is set to Never:

You have now generated an access token that never expires, which we can use to fetch leads programmatically and indefinitely, without having to bother with re-generating an access token ever again for this purpose.

Setting Up the Node.js Server

Now, the easy part: let's set up a basic web server to receive the webhook requests from Facebook every time a new lead submits their information.

1. Install Express (web server), Axios (outgoing HTTP request library) and Body Parser (for parsing requests with JSON):

npm install express axios body-parser --save  

     2. Create a file server.js and paste in the contents of this Gist.

     3. Paste the Page Access Token from the previous step into the FACEBOOK_PAGE_ACCESS_TOKEN variable.

     4. Run the server:

node server.js  

     5. Let's test out the webhook functionality with ngrok, which will create a publicly-accessible hostname that will tunnel traffic to your local Node app without having to deploy it remotely, by running:

npx ngrox http 3000  

     6. Copy the https forwarding address from the output:

Enabling the Lead Gen Webhook

1. Visit the Facebook Developer Center, select your app, scroll down to Webhooks and click Set Up.

Callback URL: Paste in the https forwarding address from ngrok, followed by /webhook

Verify Token: Enter CUSTOM_WEBHOOK_VERIFY_TOKEN (you may change this string for security purposes, but also remember to modify it in server.js accordingly).

    3. Click Verify and Save.

    4. Search for the leadgen webhook, and click Test next to it, followed by Send to My Server. Observe the terminal which is running server.js for the following error output:

This error is expected in this case, as Facebook has sent us an invalid Lead ID (444444444444). For now, ignore the error and click Subscribe to receive leadgen webhook events.

    3. We now need to manually subscribe for the leadgen webhook event for the specific page we are advertising. Execute the following curl command in your terminal of choice, replacing INSERT_PAGE_ID with your Page ID (from your Facebook Page URL) and INSERT_PAGE_ACCESS_TOKEN with the Page Access Token (which you generated previously).

curl -X POST 'https://graph.facebook.com/v2.5/INSERT_PAGE_ID/subscribed_apps?access_token=INSERT_PAGE_ACCESS_TOKEN' \
  -H 'Content-Type: application/json' \
  -d 'subscribed_fields=leadgen'

Check for a {"success": true} response.

Time to Test

Finally, let's make sure everything works!

Facebook provides a nifty Lead Ads Testing Tool which will invoke our webhook with a test lead. Select your Page and Form from the respective dropdown and click Create lead. If all went well, the Node.js console output should show the lead details!

Next Steps

To make this script actually notify you when a new lead submits their information, a final step is required to send out an e-mail or some other kind of notification to alert the relevant salesperson of the new lead, with the information entered from the form. I recommend using nodemailer for this, and AWS SES as the SMTP e-mail sending service. Check the sample code at the end of server.js for the nodemailer approach.

Finally, when you actually deploy this node app on a server, make sure to update the Callback URL in the Webhook Settings for your Facebook App to the server's address instead of the temporary ngrok address.

Let me know if you found this useful, and have any questions :)

There are several well-documented approaches to load balancing large amounts of traffic to your service. The most common involves using nginx or apache as a reverse-proxy to load-balance connections in a round-robin or least-concurrent-connections fashion.

Another common method is to use a load-balancing service such as AWS Elastic Load Balancer. Behind the scenes, AWS runs servers that accept connections from your clients and forward them to your own backend servers for processing. The AWS service tracks various metrics behind-the-scenes to decide which backend server to forward incoming requests to. Based on the docs, with Classic Load Balancers, the node that receives the request selects a registered instance as follows:

• It uses the round robin routing algorithm for TCP listeners
• It uses the least outstanding requests routing algorithm for HTTP and HTTPS listeners

While this approach to load-balancing can work seamlessly for generally short-lived connections such as HTTP requests, with long-lived TCP sockets, a round-robin routing algorithm may not balance traffic equally among your backend instances in some scenarios, such as when you spin up a new backend instance as you scale. In this scenario, it could take a while until the new instance is hosting the same number of long-lived TCP connections as the others, which may have already reached their connection limit.

Another drawback to using a service such as AWS ELB for load-balancing long-lived TCP connections is the added cost and compute power for routing these connections. With AWS ELB, each client connection is kept open as it is being load-balanced to your own backend servers for processing. Therefore, each client connection to your service essentially creates two connections: one at the ELB level, and one on your own servers. Furthermore, AWS ELB can become costly, as it also charges based on the
amount of data transferred through the load balancer to your backend instances.

Enter Dynamic DNS Load Balancing

Dynamic DNS load balancing can solve both these drawbacks, quite seamlessly. With traditional DNS mentality, we're used to defining a static host file which maps various records to static endpoints (IPs/hostnames/etc). But what if we could control the DNS resolver response on a per-request level? We could essentially load-balance traffic at the DNS level, routing each client to a backend instance based on our own criteria, such as routing to the server with the least number of established connections.

We face two obstacles:

1) Most DNS services (including AWS Route 53) don't support implementing your own routing logic, nor executing dynamic code before returning a DNS response.

2) Client machines and ISPs like to cache a DNS response for a given hostname, sometimes for longer than the TTL value on that DNS record.

Setting up a Dynamic DNS Server

The first obstacle is pretty easy to solve with Node.js -- it turns out there's plenty of DNS server implementations, which let you handle each client request individually, and return a dynamic, load-balanced response, instead of a fixed one.

Unfortunately, most of the Node.js packages I tested were either outdated or broken, returning invalid DNS responses (extra/missing bytes in the response) or not supporting the common record types, or error codes such as REFUSED or NXDOMAIN.

Finally, I came across mname which validated across most DNS server tests I performed and allowed for per-request response processing:

var named = require('mname');  
var server = named.createServer();

// In production, run the server on TCP/UDP port 53\
// or use iptables to forward traffic
var port = 9000;

// Listen on TCP
server.listenTcp({ port: port, address: '::' });

// Listen on UDP
server.listen(port, '::', function() {  
  console.log('DNS server started on TCP/UDP port ' + port);
});

server.on('query', function(query, done) {  
  // Extract query hostname and set default TTL to 60 seconds
  var name = query.name(), ttl = 60;

  // Log incoming DNS query for debugging purposes
  console.log('[DNS] %s IN %s', query.name(), query.type());

  // Your backend IPs
  var serverIPs = ['1.2.3.4', '8.8.4.4', '8.8.8.8'];

  // Select one randomly (modify based on your own routing algorithm)
  var result = serverIPs[Math.floor(Math.random() * serverIPs.length)];

  // Load-balance DNS queries (A record) for "api.example.com"
  if (query.type() === 'A' && name.toLowerCase().endsWith('api.example.com')) {
     // Respond with load-balanced IP address
     query.addAnswer(name, new named.ARecord(result), ttl);
  }
  else {
    // NXDOMAIN response code (unsupported query name/type)
    query.setError('NXDOMAIN');
  }

  // Send response back to client
  server.send(query);
});

Run the sample code (after running npm install mname) and you have got yourself a dynamic DNS server running in Node.js.

Go ahead and test it out with dig:

dig @127.0.0.1 -p 9000 api.example.com  

The result should look as follows:

If you run the dig command multiple times, you'll notice a random result returned for every query.

Now, all you need to do is write custom code to return the IP address of one of your backend servers to handle the client's request/connection, based on your own load balancing logic.

For example, you might maintain a database table with each backend server and its current number of established connections. Your DNS server should regularly query this table and pick the endpoint that recently reported the least number of established connections.

Pro tip: It is super important to call .toLowerCase() on the incoming query.name() when validating it against api.example.com, as some DNS clients / ISPs like to use mixed case to query your DNS server for added DNS security, which means api.example.com can become aPi.ExAmPLE.coM. But do make sure to return the mixed case query name back to the requesting client in your DNS response answer.

And just like that, you have implemented yourself a dynamic DNS load balancer. But how do we work around the unforgiving DNS caching issue performed by ISPs?

An Anti-cache Workaround

We know that multiple queries for the same hostname, api.example.com, may result in caching, thereby hindering our efforts to load-balance every single connection to our service. A clever way to work around this is by placing a random number, or the current Unix timestamp, as a prefix to our hostname:

1596207957-api.example.com  

If clients try to connect to a new hostname every time they want to establish a connection, it would never have been cached previously, and therefore your DNS service can return an uncached, fresh response, every single time.

The sample code already supports this anti-cache mechanism using the .endsWith() function:

if (name.toLowerCase().endsWith('api.example.com')) {}  

Now all that's left is to implement your routing algorithm and deploy this DNS service as a cluster. When deploying a DNS cluster, it is recommended to run 4 DNS servers with different IP addresses, each listening on TCP/UDP port 53 for incoming DNS requests, and assign each of them a hostname in another domain, such as ns1.example.io, ns2.example.io, and so on.

Refer to my Deploy Resilient Node.js Apps with Forever (skip the nginx part) to run your Node.js DNS server with forever, and refer to my Binding a Node.js App to Port 80 guide (specifically, the iptables section), tweaking the command to reroute traffic on incoming port 53 to your DNS server listening on port 9000:

sudo iptables -A PREROUTING -t nat -i eth0 -p tcp --dport 53 -j REDIRECT --to-port 9000  
sudo iptables -A PREROUTING -t nat -i eth0 -p udp --dport 53 -j REDIRECT --to-port 9000  

For your DNS servers to actually start routing queries, you need to set the target domain name's nameservers (such as as example.com) in the domain name control panel to those 4 servers' hostnames so that clients start querying your DNS servers when trying to resolve api.example.com.

Hope you found this useful, and let me know if you have any questions in the comments!

There I was, staring at yet another empty shelf in yet another pharmacy, as I was attempting to buy a digital thermometer and some fever reducing medicine for my family. Being told time and time again by store reps that these essential items are currently out of stock and I should look elsewhere.

Essentials stock shortages have unfortunately become a reality for most of us during the COVID-19 / Coronavirus pandemic. Emergency essentials we desperately need have gone out-of-stock in many stores due to panic buying, hoarding, and the sudden surge in demand.

Yet there are still stores with these essentials in stock in our cities. But visiting every single pharmacy & supermarket in a city to check if an item is in stock is a tedious, if not impossible task that may contribute to further contagion of Coronavirus.

There has to be a better way.

Meet Amy 🤖

Amy is an A.I. virtual assistant (think Google Duplex) that calls up pharmacies & supermarkets in a given city, asking store reps to confirm stock of essentials, such as:

1. Pain relief / fever reducing medicine
2. Cough medicine
3. Digital thermometers
4. Hand sanitizer
5. Antibacterial soap
6. Bottled water
7. Canned food
8. Pasta/rice

Listen to 1 of 1,000 real calls Amy 🤖 made in London to confirm stock:
https://go.aws/2JqVDLB

Here's a transcript of that call:

Store Rep: Hello Sainsbury's local Stannis speaking how can I help?

Amy: Hi. In an effort to help citizens stock up on emergency supplies during the Coronavirus pandemic, we need your help to check the availability of some items at your store.

Amy: Please answer the following questions with a yes or no.

Amy: Do you still have pain medication such as ibuprofen in stock?

Store Rep: No

Amy: Do you have cough medication?

Store Rep: No ...

Once the stock status from various stores in a city has been collected, it is made available on the InStock website.

InStock

On instock.app, visitors are presented with a map of pharmacies and supermarkets that have recently confirmed essentials in stock in their area:

Stores that stock essentials are highlighted in green, where stores without any items are greyed out.

Clicking the stores reveals exactly which items are in/out of stock, as well as when the stock status was last updated, and directions to get to this store:

Support this initiative

InStock is currently live in London, and I need your support to expand to more cities with essentials shortages. Calling up thousands of pharmacies and supermarkets in every major city on a daily basis incurs large costs, and these costs only grow with the number of cities supported.

If you are in the position to do so, please donate to support this venture:
https://paypal.me/instockapp

AWS has (quite a while ago) released EC2 instance families (such as c5, m5, and t3) that run a customized version of the Linux kernel, with some AWS goodies and performance improvements baked in, namely ena (Enhanced Networking) and nvme (Non-Volatile Memory).

If you ever tried to change the instance type of your old Ubuntu instances to one of these new instance families, you were probably presented with an error that your instance does not support Enhanced Networking:

The AWS documentation seems to cover this topic quite well at first glance, but soon enough I noticed that there seems to be a fundamental mistake in the command mentioned in the docs for enabling ENA support:

sudo apt-get upgrade -y linux-aws   # Don't run this  

It seems the AWS team was under the impression that running this command would only update/install a single package called linux-aws. In fact, quite to the contrary, running sudo apt-get upgrade will upgrade every single installed package on your machine to its latest version. This could potentially break your server and application especailly if you aren't prepared for it. Plus, there isn't a reason to suggest updating all the packages if all we're wanting to do is add ENA support.

The reality is that it's possible to enable Enhanced Networking and nvme support on your old Ubuntu instance by simply installing the linux-aws package as follows:

sudo apt-get install linux-aws -y  

Note: Please create a snapshot / AMI of your instance before running any of the commands on this page.

Then, modify the initramfs module list by running this command:

sudo nano /etc/initramfs-tools/modules  

Add nvme to the bottom of this file and save.

Finally, run the following command to update the initramfs:

sudo update-initramfs -u -k all  

Reboot your system for changes to take effect:

sudo reboot  

If you'd like to check whether both ena and nvme are successfully installed and loaded, you can use the following AWS provided script:

wget https://gist.githubusercontent.com/eladnava/1249b10c9f90c144f0d6a0fe01d93066/raw/3a82db14b90273a0b6e024114fe3ce8c4d0e345f/c5_m5_checks_script.sh  
chmod +x c5_m5_checks_script.sh  
sudo ./c5_m5_checks_script.sh  

Output should be identical to the following:

------------------------------------------------

OK     NVMe Module is installed and available on your instance


OK     ENA Module with version 2.0.3K is installed and available on your instance


OK     fstab file looks fine and does not contain any device names. 

------------------------------------------------

If indeed the output is identical, you are now ready to shut down your instance and enable the ena-support flag using the aws-cli.

• Stop the instance in the AWS console
• Get the instance ID from the AWS console
• Plug it into the following command (replace i-ABCDEFG with your instance ID)

aws ec2 modify-instance-attribute --instance-id i-ABCDEFG --ena-support  

If you have not yet installed/configured the aws-cli, please refer to these instructions.

Once this command executes without any error, you are now ready to modify your instance type to the desired ENA-enabled instance family, such as c5, m5 or t3, in the AWS EC2 console.

That's it! Hope I helped avoid a catastrophe and get your old-but-reliable instances running on a new instance family.

The original Google Pixel is an awesome phone. The Google Pixel 2? A little less, if you ask me. It's no secret that the XL version is plagued with a plethora of problems: a blue tint on the display and permanent pixel ghosting, to name a few. The pixel ghosting was improved with a software update (or hack, actually, as a workaround for the hardware issue). And the non-XL version is just plain ugly for this day and age, especially for its price tag.

Which is why I stuck with the first Google Pixel, and I'm loving it. The camera sensor is absolutely spectacular and some shots are almost indistinguishable from ones taken with a fancy DSLR:

Yes, the image has been edited, but you still need a great camera sensor for it to look this good, and the Pixel delivers.

The Issue at Hand

One thing bothers me so much about the Pixel that I decide to spend hours on a workaround: the infamous Android File Transfer bug. This is a bug in the file transfer utility that Google has provided for macOS users which makes it impossible to manage files on your Google Pixel, causing most transfers to/from the phone to fail or freeze. And Google apparently does not seem to care much ([1] [2]).

Why does Google even need to provide a file transfer utility for Android devices on macOS? Probably because Apple is reluctant to provide drivers for Android devices out of the box, complicating Android file management and possibly improving iOS device adoption rates due to the additional step involved in managing files on your Android phone.

This bug is indeed only experienced by some people that use a Mac to manage their Pixel's files. The exact manifestation requirements are still unknown, but the Internet is filled with people complaining about the issue.

Possible Workarounds

Naturally, I start to investigate some possible workarounds. One is to use an app like AirDrop to wirelessly transfer files over a Wi-Fi network. However, that app requires a vast array of prying permissions for its other features, such as display mirroring, and most importantly, transfer speeds are quite slow in comparison to a wired USB-C transfer.

Another option is to use the USB-to-USB-C adapter that Google ships with Google Pixel phones. This apparently works around the issue in Android File Transfer and transfers work fine. But that requires your Mac to have a USB-C input, which mine does not, having acquired it in early 2015. And it requires you to carry that dongle everywhere, and make sure you don't ever lose it.

The final option that comes to mind is using ADB, the Android Debug Bridge. ADB comes with useful commands like adb push and adb pull which can be used to both upload and download files to/from your device. This approach successfully works around the bug present in Android File Transfer, but it's not very convenient, as you have to manage your files via the CLI, typing commands in the macOS Terminal. This, in my opinion, is not a proper solution in the long term, especially if you regularly pull photos off of your device, or load new music, for example.

Pixelmate

And thus, Pixelmate was born. Pixelmate is an open-source macOS app I built using Electron that issues ADB commands behind the scenes to manage files on your Google Pixel device.

Pixelmate is designed to look and feel like the native macOS Finder for a familiar user experience:

Follow these instructions to download Pixelmate and manage your files like a boss.

Note: Since Pixelmate uses ADB behind the scenes, you need to enable Developer Mode and USB Debugging on your phone, if you haven't already. Check out this guide for detailed instructions.

You can drag files and folders into Pixelmate to upload them to your device, or right click remote files and folders to download them to your computer. I really wanted to implement dragging and dropping remote files to the local computer as well, but this is actually not possible at this time due to Electron platform limitations. I'm waiting on the Electron team to provide a solution to the issue.

Navigation is currently done via the same keyboard shortcuts as in Finder:

• Cmd + ↓ to navigate into a folder
• Cmd + ↑ to navigate out of a folder
• Cmd + Delete to delete a file or folder

UI navigation buttons will surely be added in the future.

Pro tip: You can even use Pixelmate wirelessly, without having to connect your phone via USB cable. Check out this guide for detailed instructions, and check out this app I built to avoid having to connect the phone to run adb tcpip before going wireless. Note that transfers will not be as fast as with a wired connection, though.

Feedback

I'd love to hear what you think of Pixelmate! For me, it's a lifesaver. Let me know in the comments below!

If you are an avid reader of mine, you might have noticed that I haven't posted for quite some time (almost a year!). Time sure flies when you're having fun. I'll do my best to post much more in 2018.

And now, let's get to the matter at hand.

Motivation

Have you ever needed to scale your application servers using a custom metric, such as available system memory or concurrent connections count?

Some application servers need to be scaled when memory becomes a bottleneck as each client adds to the application's memory utilization, and in other cases, applications can only support a finite number of concurrent socket connections before reaching their limit.

It still surprises me that AWS CloudWatch does not provide metrics for monitoring EC2 servers' memory utilization. It seems so trivial, especially since other metrics such as CPU Utilization, Disk I/O, and Network I/O are readily-available. Also, a memory metric would make monitoring your servers for memory leaks so much easier, instead of finding out about leaks after the out-of-memory killer terminates your app, causing downtime.

It would have been great if AWS provided more metrics out of the box.

But no matter, that's where Scalemate comes in! (Get it? stalemate; play on words!).

Scalemate

Scalemate is a Node.js CLI package I built that scales your application servers by publishing custom system metrics to AWS CloudWatch. The following custom metrics are currently supported:

• Sockets Used - number of active client/server connections
• Memory Available - amount of system memory available (in mb)

In addition, Scalemate supports per-second metric resolution for scaling your cluster within seconds in response to high demand.

Usage

Using Scalemate is super easy. Simply install Node.js on one of the servers in your cluster and then install Scalemate using npm:

sudo npm install -g scalemate  

Then, create a file called scalemate.js in /etc:

sudo nano /etc/scalemate.js  

Paste in the following contents:

module.exports = {  
    // Metrics to publish
    metrics: {
        // Number of open socket connections
        socketsUsed: {
            // Whether to publish this metric
            enabled: true,
            // CloudWatch unit type
            unit: 'Count',
            // CloudWatch metric title
            name: 'Sockets Used'
        },
        // Number of megabytes of system memory currently available
        memoryAvailable: {
            // Whether to publish this metric
            enabled: true,
            // CloudWatch unit type
            unit: 'Count',
            // CloudWatch metric title
            name: 'Memory Available'
        }
    },
    // Metric interval (in seconds)
    interval: 60,
    // CloudWatch namespace to associate metrics with
    namespace: 'MyApp',
    // AWS IAM user with CloudWatch read/write access
    credentials: {
        region: 'us-east-1',
        accessKeyId: 'ABCDEFG',
        secretAccessKey: 'ABCDEFGHIJK/HIJKLMNOPQRS'
    }
};

Modify the configuration according to your own needs, enabling or disabling metrics and configuring the following parameters:

• namespace - a title for your app or server cluster
• credentials - an AWS IAM user with read/write access to CloudWatch

You can create an IAM user in the AWS Security Credentials console.

Make sure to grant your IAM user the CloudWatchFullAccess policy for read/write access to CloudWatch.

Testing

Test the configuration you created by running:

scalemate -c /etc/scalemate.js  

Observe the terminal output for any initial errors and for successfully-published metrics. If no errors are emitted, you have successfully configured Scalemate!

Verification

Visit the CloudWatch console and find the published metrics under the Scalemate namespace.

Select your app namespace and you should be able to see the custom metrics you configured!

Surviving Reboots

To start Scalemate automatically after system reboots, edit your user's crontab by running:

crontab -e  

Then, append the following line to the end of the crontab:

@reboot /usr/bin/scalemate -c /etc/scalemate.js 2> /tmp/scalemate.log &

Save and reboot, then, verify that Scalemate is running:

ps aux | grep scalemate  

Finally, create an image of the server you installed and configured Scalemate on, and configure your entire EC2 cluster to use the same image. That way, the entire cluster will be publishing these custom metrics to AWS CloudWatch.

CloudWatch will, in turn, average out the metrics reported by all the servers in your cluster and let you define scaling alarms based on metric average values.

Scaling Your Cluster

Congratulations, you can now configure CloudWatch alarms in the AWS CloudWatch console based on these custom metrics!

Simply edit the existing CloudWatch alarms for your Auto Scaling Group and modify the metric being monitored, selecting one of the custom Scalemate metrics and defining an applicable alarm threshold based on the metrics.

Have any suggestions on additional custom metrics that should be added to Scalemate? Let me know in the comments! =)

You've just finished working on your shiny new JavaScript project, after months of hacking away at it, living on nothing but granola bars and instant ramen noodles, and making use of hundreds of npm dependencies. The JavaScript ecosystem is great in that sense, where a package exists for almost everything you want to achieve, and reinventing the wheel is not usually necessary.

However, this comes at a price. The more dependencies you rely upon in your projects, the higher the chance one of those dependencies, or one of its dependencies, has a restrictive license that requires you to fulfill some unusual obligation.

How unusual?

Did you know that some open source software licenses require you to disclose your source code in its entirety if you use a package with such a license, such as the GPL-2.0 and AFL-3.0 licenses?

Or that there are software licenses that require you to explicitly mention the software in all of your product's advertising materials, such as the original BSD 4-Clause license?

Some of these obligations are not easily met for commercial projects, which are usually closed source. Every organization has these, yes, even GitHub and npm do not open source all of their code.

Chances are, if your project has over 15 dependencies, at least one of their dependencies or their dependencies' dependencies is using a restrictive license with unusual obligations. If you don't check thoroughly and fulfill such obligations, you're susceptible to legal action by the package author(s), even if your project is free to use and open sourced.

Now, if you were to commercially distribute your project using a dependency with an unmet obligation, and that third party were to find out about it, well, let's hope that never happens.

You can easily prevent this from ever happening by using a new tool I released called tldrlegal.

tldrlegal

tldrlegal is a Node.js command-line tool that checks your dependencies for license requirements using a legal resource called tldrlegal.com, which provides plain English software license interpretations.

tldrlegal makes use of legally, a Node.js package that does an excellent job at determining your dependencies' licenses, using their package.json file, the README.md file, and the LICENSE file, since package maintainers use either of those to mention their license of choice. It turns out this is not the easiest of tasks, but legally still manages to do it with great accuracy.

tldrlegal.com lets you find pretty much any popular software license, and be able to quickly understand what you can and can't do with that license, as well as what you must do if you make use of software with such license. Without this website, tldrlegal could not exist.

How to use tldrlegal

Simply install tldrlegal globally via npm and run it in your project directory. The output will contain a summary and detailed information for each package with a licensing requirement, such as credit attribution, source disclosure, etc.

npm install -g tldrlegal

cd my-js-project  
tldrlegal  

If any license restrictions are found, tldrlegal will output them to the console, along with a brief description:

That's it, let me know what you think and if you have any ideas on how to improve tldrlegal!

Disclaimer

No legal-advising tool is ever complete without a proper disclaimer.

1. This tool is not a replacement for proper legal consultation.
2. Please be advised that the information provided by this tool may not be 100% accurate.

CocoaPods is the most popular dependency manager for Swift and Objective-C Cocoa projects, but chances are, you already knew that if you're here, wanting to publish your own CocoaPod to be used by others in their own projects.

Publishing a standard, open-source CocoaPod is relatively straightforward -- lots of tutorials are widely-available that outline the process rather efficiently -- but there lacks a definitive guide on how to publish a universal, binary CocoaPod: one that does not disclose its source files and supports both physical iOS device architectures (armv7, arm64) and virtual iOS simulator architectures (i386, x86_64).

Sometimes, you are simply not at liberty to disclose the source code of a CocoaPod. You might work for a company that develops an SDK which, for competitive reasons, must not be open source. That's when distributing a universal, binary framework is a must.

Install CocoaPods

Obviously, to create a pod, you need to install CocoaPods.

For Xcode 8, you'll need CocoaPods version 1.1.0.rc.3 or newer. If you already have CocoaPods installed (check by running pod --version) and its version is older than the aforementioned version, first uninstall it by running sudo gem uninstall cocoapods.

To install the latest version of CocoaPods, execute the following command:

sudo gem install cocoapods --pre  

Create a Project

Create a new Xcode project for your framework and select the Cocoa Touch Framework template:

Enter a Product Name and choose Swift as the project language. For the purpose of writing this guide, I've chosen to create a framework called MySDK that exposes a method which simply prints a String to the console.

Feel free to replace this dummy implementation with your own functionality, and replace MySDK with your own framework name whenever mentioned in the rest of this guide.

After Xcode finishes creating the project, feel free to delete the MySDK.h file included in the template, as you won't be needing it.

Develop Functionality

Time to actually write and expose some APIs in your framework.

Create a file called MySDK.swift within the MySDK group, with the following contents:

import Foundation

public class MySDK {  
    public class func logToConsole(msg: String) {
        print(msg);
    }
}

Note that you must explicitly label all classes and methods you wish to expose in your framework with the public keyword, otherwise, they won't be accessible in other projects.

Create a Demo Project

To test the framework in action and make sure it works as expected, we'll create a demo iOS app within the framework project that depends on the framework and invokes its method(s).

Open the project editor for the MySDK target and click on Editor -> Add Target in the menu bar.

Select Single View Application as the template:

For the product name, enter Demo, and select Swift as the project language.

When the demo target is created, navigate to its project editor, scroll down to the Embedded Binaries section, click the + icon, and select MySDK.framework:

Interface with the Framework

Open the demo project's AppDelegate.swift file and import your framework:

import MySDK  

In the applicationDidBecomeActive method, add the following code to log that the app is now active:

MySDK.logToConsole(msg: "Application active")  

And in the applicationDidEnterBackground method, add a different message:

MySDK.logToConsole(msg: "Application inactive")  

Run the demo project on your iOS device or simulator. The Xcode console should print Application active once the app finishes launching.

Press the Home button (Cmd + Shift + H on the iOS Simulator) and the console should print Application inactive.

It appears that the demo project is able to successfully interface with the framework!

Enable Archiving

Cocoa Touch Frameworks, by default, cannot be archived.

You can enable your framework to be archived by editing the framework target's Build Settings and setting Skip Install to No:

However, if you attempt to archive now, Xcode will only build the armv7 and arm64 executables, which would make it impossible to run the framework on the iOS simulator.

Generate Universal Framework

Due to a bug in Xcode, it is impossible to archive universal frameworks without relying on external scripts. To include binaries for the iOS simulator, you can configure a post-archive script that will build your framework for the iOS simulator after you archive it, and merge both the simulator and iOS binaries into one fat universal framework.

To have Xcode run the script after archiving automatically, select the MySDK target, click Product -> Scheme -> Edit Scheme (or Cmd + Shift + <), and configure a Run Script post-action for the Archive command:

Copy this script and paste it into the Run Script window (thanks to @atsepkov for the original script).

Be sure to select MySDK for the Provide build settings from setting and click Close to apply the changes.

Archive Framework

Finally, archive the framework by clicking Build -> Archive in the menu bar. If the option is greyed out, make sure to select a physical iOS device and not the iOS simulator.

Once the bundle is archived, the Xcode Organizer will pop up. Wait a few more seconds, and the Finder should also open up to your project directory with a universal MySDK.framework inside it.

You can verify that MySDK.framework is indeed a universal framework by running file MySDK within the MySDK.framework directory:

As the output suggests, the executable contains binaries for the i386, x86_64, armv7 and arm64 architectures, which makes it a universal, fat framework.

CocoaPod Specifications

Now that you have successfully exported the universal framework, let's distribute it as a CocoaPod!

Create a MySDK.podspec file in your project directory which will contain information about the CocoaPod you are publishing, such as its name, version, sources, and more.

Paste the following contents inside it:

Pod::Spec.new do |s|  
    s.name              = 'MySDK'
    s.version           = '1.0.0'
    s.summary           = 'A really cool SDK that logs stuff.'
    s.homepage          = 'http://example.com/'

    s.author            = { 'Name' => 'sdk@example.com' }
    s.license           = { :type => 'Apache-2.0', :file => 'LICENSE' }

    s.platform          = :ios
    s.source            = { :http => 'http://example.com/sdk/1.0.0/MySDK.zip' }

    s.ios.deployment_target = '8.0'
    s.ios.vendored_frameworks = 'MySDK.framework'
end  

Feel free to customize the .podspec to your liking. Here is an explanation for some of the more non-trivial properties:

• s.license - you must ship a license file with your CocoaPod, so go ahead and create a LICENSE file in your project directory with this as its content.
• s.source - the hosted .zip location of your CocoaPod files (in your case, the MySDK.framework folder and the LICENSE file). More on this later.
• s.ios.vendored_frameworks - the path of the framework you are distributing within the s.source archive, after being decompressed.

Did you know? When you publish a CocoaPod, the only file that actually gets pushed up to the CocoaPods repository is your .podspec file. Nothing else, which is why you must link to the hosted files via the s.source parameter.

Create a Zip Archive

The easiest way to make your framework and license available with your CocoaPod is to archive them inside a .zip, upload it to your server, and link to it using the s.source parameter in the .podspec file.

Create the MySDK.zip file by running the following command in your project directory:

zip -r MySDK.zip LICENSE MySDK.framework  

It's up to you to upload the MySDK.zip file to your server, in a path similar to the following:

http://example.com/sdk/1.0.0/MySDK.zip  

You can also create a GitHub repository and push the .zip file to it.

Once you upload the .zip, link to it in the s.source parameter of the .podspec file.

Pre-Publish Testing

Before you publish, you will want to make sure that the CocoaPod .podspec is correctly configured to distribute your framework.

Create a brand new Xcode project and select Single View Application as its template.

Run the following command in the project directory:

pod init  

Edit the Podfile and paste the following within the target declaration to reference a CocoaPod dependency from the local filesystem:

pod 'MySDK', :podspec => '/Code/mysdk-pod/'  

Modify the :podspec to the local path of the framework project containing the MySDK.podspec file.

Save the file and run the following command to install the CocoaPod:

pod install  

Reopen the test project via its newly-generated .xcworkspace file, and attempt to import your framework within the AppDelegate.swift file:

import MySDK  

Also, attempt to access the various method(s), such as MySDK.logToConsole. Run the app on an iOS device. If everything worked as expected, you're ready to publish your framework!

Register a Trunk Account

To publish your .podspec file to the CocoaPods repository, you must first register an account with the CocoaPods Trunk.

The CocoaPods Trunk is an authentication and CocoaPods API service. To publish new or updated libraries to CocoaPods for public release, you will need to be registered with the Trunk and have a valid Trunk session on your current device.

Register an account by running the following, entering your full name and e-mail address:

pod trunk register you@email.com 'Full Name'  

Then, check your e-mail for a confirmation link and click it.

Your Trunk account is now activated and you can finally publish your CocoaPod!

Publish the Pod

Run the following command in the same directory as the .podspec to publish it to the CocoaPods repository:

pod trunk push MySDK.podspec  

The CLI will validate your .podspec and attempt to install the CocoaPod by downloading the source .zip and validating its contents. If the command succeeds, you have just published your first universal binary CocoaPod!

Test Again

Create another test project, run pod init, and add the following to the Podfile:

pod 'MySDK', '1.0.0'  

Then, run pod install and cross your fingers.

Reopen the test project's .xcworkspace file and once again, attempt to interface with your framework. Run the app on an iOS device. If everything worked as expected, you should be good to go!

Let me know if this guide helped you in the comments below!

Push notifications are a great way to ensure your users re-engage with your app every once in a while, but implementing them on iOS can be challenging, especially with all of the changes in Xcode and Swift, not to mention the various iOS versions which deprecate the notification classes we grew accustomed to in the past.

The Internet is overflowing with guides on how to implement iOS push notifications -- however, many of these guides are cumbersome, complicated, not up-to-date with Swift 3 and Xcode 8, and/or don't provide backward-compatibility with all iOS versions that support Swift (iOS 7 - iOS 10). Also, they do not make use of the new APNs Auth Keys which greatly simplify the steps involved in sending push notifications.

By following this guide, you'll be able to implement push notifications in your iOS app and send notifications from Node.js, using the latest technologies and without much hassle!

Preparations

First off, open your iOS project in Xcode 8. If you don't have Xcode 8 yet, be sure to update via the App Store. If you don't have an iOS project yet, simply create a new one. Make sure that your codebase has been updated to use Swift 3.

Second, make sure that you have an active Apple Developer Program Membership, which costs $100/year. It is a requirement in order to send push notifications to your iOS app. Also, make sure Xcode is configured to use the iCloud account which contains your active Apple Developer Program Membership.

Third, make sure that your app has a Bundle Identifier configured in the project editor:

Enabling Push Notifications

The first step in setting up push notifications is enabling the feature within Xcode 8 for your app. Simply go to the project editor for your target and then click on the Capabilities tab. Look for Push Notifications and toggle its value to ON:

Xcode should display two checkmarks indicating that the capability was successfully enabled. Behind the scenes, Xcode creates an App ID in the Developer Center and enables the Push Notifications service for your app.

Registering Devices

Devices need to be uniquely identified to receive push notifications.

Every device that installs your app is assigned a unique device token by APNs that you can use to push it at any given time. Once the device has been assigned a unique token, it should be persisted in your backend database.

A sample device token looks like this:

5311839E985FA01B56E7AD74334C0137F7D6AF71A22745D0FB50DED665E0E882  

To request a device token for the current device, open AppDelegate.swift and add the following to the didFinishLaunchingWithOptions callback function, before the return statement:

// iOS 10 support
if #available(iOS 10, *) {  
    UNUserNotificationCenter.current().requestAuthorization(options:[.badge, .alert, .sound]){ (granted, error) in }
    application.registerForRemoteNotifications()
}
// iOS 9 support
else if #available(iOS 9, *) {  
    UIApplication.shared.registerUserNotificationSettings(UIUserNotificationSettings(types: [.badge, .sound, .alert], categories: nil))
    UIApplication.shared.registerForRemoteNotifications()
}
// iOS 8 support
else if #available(iOS 8, *) {  
    UIApplication.shared.registerUserNotificationSettings(UIUserNotificationSettings(types: [.badge, .sound, .alert], categories: nil))
    UIApplication.shared.registerForRemoteNotifications()
}
// iOS 7 support
else {  
    application.registerForRemoteNotifications(matching: [.badge, .sound, .alert])
}

In iOS 10, a new framework called UserNotifications was introduced and must be imported in order to access the UNUserNotificationCenter class.

Add the following import statement to the top of AppDelegate.swift:

import UserNotifications  

Next, go to the project editor for your target, and in the General tab, look for the Linked Frameworks and Libraries section.

Click + and select UserNotifications.framework:

Next, add the following callbacks in AppDelegate.swift which will be invoked when APNs has either successfully registered or failed registering the device to receive notifications:

// Called when APNs has assigned the device a unique token
func application(_ application: UIApplication, didRegisterForRemoteNotificationsWithDeviceToken deviceToken: Data) {  
    // Convert token to string
    let deviceTokenString = deviceToken.reduce("", {$0 + String(format: "%02X", $1)})

    // Print it to console
    print("APNs device token: \(deviceTokenString)")

    // Persist it in your backend in case it's new
}

// Called when APNs failed to register the device for push notifications
func application(_ application: UIApplication, didFailToRegisterForRemoteNotificationsWithError error: Error) {  
    // Print the error to console (you should alert the user that registration failed)
    print("APNs registration failed: \(error)")
}

It's up to you to implement logic that will persist the token in your application backend. Later in this guide, your backend server will connect to APNs and send push notifications by providing this very same device token to indicate which device(s) should receive the notification.

Note that the device token may change in the future due to various reasons, so use NSUserDefaults, a local key-value store, to persist the token locally and only update your backend when the token has changed, to avoid unnecessary requests.

Run your app on a physical iOS device (the iOS simulator cannot receive notifications) after making the necessary modifications to AppDelegate.swift. Look for the following dialog, and press OK to permit your app to receive push notifications:

Within a second or two, the Xcode console should display your device's unique token. Copy it and save it for later.

Prepare to Receive Notifications

Add the following callback in AppDelegate.swift which will be invoked when your app receives a push notification sent by your backend server:

// Push notification received
func application(_ application: UIApplication, didReceiveRemoteNotification data: [AnyHashable : Any]) {  
    // Print notification payload data
    print("Push notification received: \(data)")
}

Note that this callback will only be invoked whenever the user has either clicked or swiped to interact with your push notification from the lock screen / Notification Center, or if your app was open when the push notification was received by the device.

It's up to you to develop the actual logic that gets executed when a notification is interacted with. For example, if you have a messenger app, a "new message" push notification should open the relevant chat page and cause the list of messages to be updated from the server. Make use of the data object which will contain any data that you send from your application backend, such as the chat ID, in the messenger app example.

It's important to note that in the event your app is open when a push notification is received, the user will not see the notification at all, and it is up to you to notify the user in some way. This StackOverflow question lists some possible workarounds, such as displaying an in-app banner similar to the stock iOS notification banner.

Generate an APNs Auth Key

The next step involves generating an authentication key that will allow your backend server to authenticate with APNs when it wants to send one or more of your devices a push notification.

Up until a few months ago, the alternative to this was a painful process that involved filling out a Certificate Signing Request in Keychain Access, uploading it to the Developer Center, downloading a signed certificate, and exporting its private key from Keychain Access (not to mention converting both certificates to .pem format). This certificate would then expire and need to be renewed every year and would only be valid for one deployment scheme: Development or Production.

Thankfully, Apple has greatly simplified the process of authenticating with APNs with the introduction of APNs Auth Keys, which never expire (unless revoked by you) and work for all deployment schemes.

Open the Keys -> All page in your Developer Center and click the + button to create a new Auth Key.

In the next page, enter a name for your key, enable APNs and click Continue at the bottom of the page.

Finally, click Confirm in the next page. Apple will then generate a .p8 key file containing your APNs Auth Key.

Download the .p8 key file to your computer and save it for later. Also, be sure to write down the Key ID somewhere, as you'll need it later when connecting to APNs.

Send Push Notifications

Now it's time to set up your backend to connect to APNs to send notifications to devices! For the purpose of this guide and for simplicity, I'll choose to do this in Node.js. If you already have a backend implemented in another development language, look for another guide better-tailored for you, or simply follow along to send a test push notification to your device.

Make sure you have Node.js v4 or newer installed on your local machine and run the following in a directory of your choice:

mkdir apns  
cd apns  
npm init --yes  
npm install apn --save  

These commands will initiate a new Node.js project and install the amazing apn package from npm, which authenticates with APNs and sends your push notifications.

Next, copy the .p8 file you just downloaded into the apns folder we created. Name it apns.p8 for simplicity.

Create a new file in the apns folder named app.js using your favorite editor, and paste the following code inside:

var apn = require('apn');

// Set up apn with the APNs Auth Key
var apnProvider = new apn.Provider({  
     token: {
        key: 'apns.p8', // Path to the key p8 file
        keyId: 'ABCDE12345', // The Key ID of the p8 file (available at https://developer.apple.com/account/ios/certificate/key)
        teamId: 'ABCDE12345', // The Team ID of your Apple Developer Account (available at https://developer.apple.com/account/#/membership/)
    },
    production: false // Set to true if sending a notification to a production iOS app
});

// Enter the device token from the Xcode console
var deviceToken = '5311839E985FA01B56E7AD74444C0157F7F71A2745D0FB50DED665E0E882';

// Prepare a new notification
var notification = new apn.Notification();

// Specify your iOS app's Bundle ID (accessible within the project editor)
notification.topic = 'my.bundle.id';

// Set expiration to 1 hour from now (in case device is offline)
notification.expiry = Math.floor(Date.now() / 1000) + 3600;

// Set app badge indicator
notification.badge = 3;

// Play ping.aiff sound when the notification is received
notification.sound = 'ping.aiff';

// Display the following message (the actual notification text, supports emoji)
notification.alert = 'Hello World \u270C';

// Send any extra payload data with the notification which will be accessible to your app in didReceiveRemoteNotification
notification.payload = {id: 123};

// Actually send the notification
apnProvider.send(notification, deviceToken).then(function(result) {  
    // Check the result for any failed devices
    console.log(result);
});

There are several things to do before running this code:

1. Configure the keyId property with the APNs Auth Key ID (available at https://developer.apple.com/account/ios/certificate/key)
2. Configure the teamId property with your Apple Developer Account Team ID (available at https://developer.apple.com/account/#/membership/)
3. Configure deviceToken with the device token you generated after running your application and checking the console
4. Configure notification.topic with your application's Bundle ID which is accessible in the project editor

Now, lock your device, run node app.js and lo-and-behold, provided you did everything right, your iOS device should be able to receive the notification!

Interacting with the notification will print the following in your Xcode console since didReceiveRemoteNotification is invoked:

[AnyHashable("id"): 123, AnyHashable("aps"): {
    alert = "Hello World \U270c";
    badge = 3;
    sound = "ping.aiff";
}]

I hope you were able to get through this tutorial with ease. Let me know if this helped you in the comments below!

Ah, MongoDB. Arguably the leading NoSQL database available today, it makes it super easy to start hacking away on projects without having to worry about table schemas, while delivering extremely fast performance and lots of useful features.

However, running a scalable, highly-available MongoDB cluster is a whole 'nother story. You need a good understanding of how replica sets work and be familiar with the inner workings of MongoDB. And you need to set up tooling to constantly monitor your cluster for replication lag, CPU usage, disk space utilization, etc, as well as periodically back up your databases to prevent data loss.

While solutions such as Compose and MongoDB Atlas could save you the time and effort required in setting up and maintaining your own cluster, these solutions give you less control in the end -- your data and uptime is in another company's hands, in addition to AWS.

I have not been fortunate with these kinds of solutions -- I experienced several unexpected instances of downtime that my services cannot tolerate. When things went wrong, all I could do was open a support ticket and wait (sometimes days!) until the engineers would be able to resolve the issue.

If the database is the single most important part of your application, leave the controls in your hands. Setting up our own cluster is actually not that hard, let's get to it!

Note: This is an extremely comprehensive guide, make sure you have at least an hour to spare.

Replica Sets

So, what is a replica set? Put simply, it is a group of MongoDB servers operating in a primary/secondary failover fashion. At any point there can only be one primary member within the replica set, however, you can have as many secondaries as you want. All secondaries actively replicate data off of the current primary member so that if it fails, one of them will be able to take over quite seamlessly as the new primary. They do this by examining the primary member's oplog, a file that contains a log of every single write query performed against the server.

The more secondaries you have, and the more spread out they are over availability zones or regions, the less chance your cluster will ever experience downtime.

Your application will usually only run queries against the primary member in the replica set.

Replica Set Members

The most minimal replica set setup must have at least three healthy members to operate. One member will serve as the primary, another as the secondary, and the third as an arbiter.

Arbiters are members that participate in elections in order to break ties and do not actually replicate any data. If a replica set has an even number of members, we must add an arbiter member to act as a tie-breaker, otherwise, when the primary member fails or steps down, a new primary will not be elected!

This requirement was put in place to prevent split-brain scenarios where 2 secondary members in a cluster who can't communicate with each other decide to vote for themselves, causing them both to become primaries, leading to data inconsistency and a plethora of other problems.

You can also avoid an arbiter by simply adding another secondary instead. All you really need is an odd number of members for elections to be held properly. However, an extra secondary will cost you more than an arbiter.

Instance Types

The data members in the replica set should be deployed to an instance type that suits your application's needs. Depending on your traffic, queries per second, and data size, you'll need to pick a matching instance type to accommodate that workload. The good news is that you can upgrade your instance type in the future in a matter of minutes and without downtime by utilizing replica set step downs, as we'll see later on.

I usually go with either m3.medium or m4.large for a production application with about 50 queries per second. If you're just starting to work on a new project, even t2.nano will do just fine. Note that t2 instances have limited CPU credits and should not be used for high-throughput deployments, since their compute capacity is unpredictable.

It is absolutely fine to host the arbiter member on a weak instance type such as t2.nano, since all it will ever do is participate in elections.

Instance Storage

Always provision General Purpose (gp2) storage for MongoDB data members as the underlying disk is a network-attached SSD which will provide better read/write speeds than magnetic storage.

Note that if you select an m4.large instance type or larger, you'll also get EBS optimization which will provide your instance with dedicated IO throughput to your EBS storage, increasing the number of queries per second your cluster will be able to handle, as well as preventing replication lag to your secondaries.

In addition, if you want to maximize your read/write IO throughput rate and your workload is big enough, consider using Provisioned IOPS (io1) storage. This can be quite expensive though, depending on the number of IOPS you provision, so make sure you understand the pricing implications.

Get Started

The first step in setting up the replica set is to prepare the instances for running MongoDB and to make sure you have your own domain name.

Provision the Instances

Spin up 3 brand-new Ubuntu 14.04 LTS instances in the EC2 console, making sure to set up each one in a different availability zone, for increased availability in case of service outage in one AZ. Provision enough storage to fit your data size, and select the appropriate instance types for each replica set member. Also, create an EC2 key pair so that you can SSH into the instances.

Create a new security group, mongodb-cluster, and configure all three instances to use it. Allow SSH on port 22 from your IP only and port 27017 from the mongodb-cluster security group (sg-65d4d11d for example) as well as from your IP address, so that both you and the replica set members will be able to connect to each other's mongod process listening on port 27017.

Next, request 3x Elastic IPs and attach them to each instance, so that your members will maintain the same public IP throughout their entire lifetime.

Finally, label each instance you created as follows, replacing example.com with your domain name:

• Data - db1.example.com
• Data - db2.example.com
• Arbiter - arbiter1.example.com

Setup DNS Records

Head over to your domain's DNS management interface and add CNAME records for db1, db2, and arbiter1. For each record, enter each instance's Public DNS hostname, visible in the EC2 instances dashboard.

Pro tip: When your EC2 servers perform a DNS query to translate the Public DNS hostname to an IP, the EC2 DNS server will actually return the private IP address of the instance since it's in the same VPC as the instance performing the DNS query, thereby improving latency and bandwidth between the replica set members, and saving you from paying bandwidth costs.

Configuring the Servers

Before we can get the replica set up and running, we need to make a few modifications to the underlying OS so that it behaves nicely with MongoDB.

Set the Hostname

SSH into each server and set its hostname so that when we initialize the replica set, members will be able to understand how to reach one another:

sudo bash -c 'echo db1.example.com > /etc/hostname && hostname -F /etc/hostname'

Make sure to modify db1.example.com and set it to each server's DNS hostname. The first command will set the server's hostname in /etc/hostname, the second will apply it without having to reboot the machine.

Repeat this step on all replica set members.

Increase OS Limits

MongoDB needs to be able to create file descriptors when clients connect and spawn a large number of processes in order to operate effectively. The default file and process limits shipped with Ubuntu are not applicable for MongoDB.

Modify them by editing the limits.conf file:

sudo nano /etc/security/limits.conf

Add the following lines to the end of the file:

* soft nofile 64000
* hard nofile 64000
* soft nproc 32000
* hard nproc 32000

Next, create a file called 90-nproc.conf in /etc/security/limits.d/:

sudo nano /etc/security/limits.d/90-nproc.conf

Paste the following lines into the file:

* soft nproc 32000
* hard nproc 32000

Repeat this step on all replica set members.

Disable Transparent Huge Pages

Transparent Huge Pages (THP) is a Linux memory management system that reduces the overhead of Translation Lookaside Buffer (TLB) lookups on machines with large amounts of memory by using larger memory pages.

However, database workloads often perform poorly with THP, because they tend to have sparse rather than contiguous memory access patterns. You should disable THP to ensure best performance with MongoDB.

Run the following commands to create an init script that will automatically disable THP on system boot:

sudo nano /etc/init.d/disable-transparent-hugepages

Paste the following inside it:

#!/bin/sh
### BEGIN INIT INFO
# Provides:          disable-transparent-hugepages
# Required-Start:    $local_fs
# Required-Stop:
# X-Start-Before:    mongod mongodb-mms-automation-agent
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Disable Linux transparent huge pages
# Description:       Disable Linux transparent huge pages, to improve
#                    database performance.
### END INIT INFO

case $1 in
  start)
    if [ -d /sys/kernel/mm/transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/transparent_hugepage
    elif [ -d /sys/kernel/mm/redhat_transparent_hugepage ]; then
      thp_path=/sys/kernel/mm/redhat_transparent_hugepage
    else
      return 0
    fi

    echo 'never' > ${thp_path}/enabled
    echo 'never' > ${thp_path}/defrag

    unset thp_path
    ;;
esac

Make it executable:

sudo chmod 755 /etc/init.d/disable-transparent-hugepages

Set it to start automatically on boot:

sudo update-rc.d disable-transparent-hugepages defaults

Repeat this step on all replica set data members.

Turn Off Core Dumps

MongoDB generates core dumps on some mongod crashes. For production environments, you should turn off core dumps since generating them can take minutes or even hours in case your workload is large.

sudo nano /etc/default/apport

Find:

enabled=1

Replace with:

enabled=0

Configure the Filesystem

Linux by default will update the last access time when files are modified. When MongoDB performs frequent writes to the filesystem, this will create unnecessary overhead and performance degradation. We can disable this feature by editing the fstab file:

sudo nano /etc/fstab

Add the noatime flag directly after defaults:

LABEL=cloudimg-rootfs   /        ext4   defaults,noatime,discard        0 0

Read Ahead Block Size

In addition, the default disk read ahead settings on EC2 are not optimized for MongoDB. The number of blocks to read ahead should be adjusted to approximately 32 blocks (or 16 KB) of data. We can achieve this by adding a crontab entry that will execute when the system boots up:

sudo crontab -e

Choose nano by pressing 2 if this is your first time editing the crontab, and then append the following to the end of the file:

@reboot /sbin/blockdev --setra 32 /dev/xvda1

Make sure that your EBS volume is mounted on /dev/xvda1. Save the file and reboot the server:

sudo reboot

Repeat this step on all replica set data members.

Verification

After rebooting, you can check whether the new hostname is in effect by running:

hostname

Check that the OS limits have been increased by running:

ulimit -u # max number of processes
ulimit -n # max number of open file descriptors

The first command should output 32000, the second 64000.

Check whether the Transparent Huge Pages feature was disabled successfully by issuing the following commands:

cat /sys/kernel/mm/transparent_hugepage/enabled
cat /sys/kernel/mm/transparent_hugepage/defrag

For both commands, the correct output resembles:

always madvise [never]

Check that noatime was successfully configured:

cat /proc/mounts | grep noatime

It should print a line similar to:

/dev/xvda1 / ext4 rw,noatime,discard,data=ordered 0 0

In addition, verify that the disk read-ahead value is correct by running:

sudo blockdev --getra /dev/xvda1

It should print 32.

Install MongoDB

Run the following commands to install the latest stable 3.4.x version of MongoDB:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 0C49F3730359A14518585931BC711F9BA15703C6
echo "deb [ arch=amd64 ] http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.4 multiverse" | sudo tee /etc/apt/sources.list.d/mongodb-org-3.4.list

sudo apt-get update
sudo apt-get install -y mongodb-org

These commands will also auto-start mongod, the MongoDB daemon. Repeat this step on all replica set members.

Configure MongoDB

Now it's time to configure MongoDB to operate in replica set mode, as well as allow remote access to the server.

sudo nano /etc/mongod.conf

Find and remove the following line entirely, or prefix it with a # to comment it out:

bindIp: 127.0.0.1

Next, find:

#replication:

Add the following below, replacing example-replica-set with a name for your replica set:

replication:
 replSetName: "example-replica-set"

Finally, restart MongoDB to apply the changes:

sudo service mongod restart

Make these modifications on all of your members, making sure to specify the same exact replica set name when configuring the other members.

Initialize the Replica Set

Connect to one of the MongoDB instances (preferably db1) to initialize the replica set and declare its members. Note that you only have to run these commands on one of the members. MongoDB will synchronize the replica set configuration to all of the other members automatically.

Connect to MongoDB via the following command:

mongo db1.example.com

Initialize the replica set:

rs.initiate()

The command will automatically add the current member as the first member of the replica set.

Add the second data member to the replica set:

rs.add("db2.example.com")

And finally, add the arbiter, making sure to pass in true as the second argument (which denotes that the member is an arbiter and not a data member).

rs.add("arbiter1.example.com", true)

Verify Replica Set Status

Take a look at the replica set status by running:

rs.status()

Inspect the members array. Look for one PRIMARY, one SECONDARY, and one ARBITER member. All members should have a health value of 1. If not, make sure the members can talk to each other on port 27017 by using telnet, for example.

Setup Log Rotation

By default, MongoDB will fill up the /var/log/mongodb/mongod.log file with gigabytes of data. It will be very hard to work with this log file if we do not set up log rotation in advance.

Install logrotate as follows:

sudo apt-get install logrotate

Configure log rotation for MongoDB:

sudo nano /etc/logrotate.d/mongod

Paste the following contents:

/var/log/mongodb/*.log {
    daily
    rotate 5
    compress
    dateext
    missingok
    notifempty
    sharedscripts
    copytruncate
    postrotate
        /bin/kill -SIGUSR1 `cat /var/lib/mongodb/mongod.lock 2> /dev/null` 2> /dev/null || true
    endscript
}

This will set up daily log rotation for mongod.log as well as send the SIGUSR1 signal to mongod when the log file is rotated so that it starts writing to the new log file.

Replica Set Administration

Now that your replica set is highly-available and healthy, let's go over how to manage it.

Connecting to Your Replica Set

To connect to any member of the replica set, simply run:

mongo db1.example.com

Replace db1.example.com with any of the replica set member hostnames.

To send queries to your replica set from your application, simply use a MongoDB driver along with the following connection string:

mongodb://db1.example.com,db2.example.com/db-name?replicaSet=example-replica-set

Make sure to replace example.com with your domain, db-name with the database you want to run queries against, and example-replica-set with the replica set name you configured in mongod.conf.

Performing Maintenance on the Replica Set

If you want to perform some kind of maintenance on a member of the replica set, make sure it's a secondary member. You do not want to shut down the primary member without stepping down and letting another secondary become the primary first.

Run the following command on the secondary member's mongo shell:

db.shutdownServer()

Feel free to reboot the instance, modify its instance type, add more storage, provision more IOPS, etc. When you're done, simply start up the server and make sure mongod is running. The secondary member will catch up to the primary by examining its oplog and replicating anything it missed during its downtime window.

One thing to note though, is that you should not shut down secondaries for too long, otherwise, the primary's oplog will be truncated and they won't be able to catch up. Not the end of the world, but this will require you to perform a full resync of the secondary member(s) which might take time.

When you're done performing maintenance on all of the secondaries in the replica set, make sure all members of the replica set are healthy and then issue the following command on the primary member's mongo shell to ask it to step down and let another secondary take its place:

rs.stepDown()

An election will then take place and the replica set members will vote for a new primary member. This can take anywhere from 10 to 50 seconds. During the election, the replica set will be unavailable for writes, since there is no primary member while voting takes place. Assuming you have an odd number of members, and there are healthy secondary members, a new primary will be elected and the replica set will be writable again.

Surviving Step Downs

Your application must be prepared to deal with step downs by queueing up the writes and reattempting them when the new primary has been elected.

This can easily be achieved with a Node.js package I developed called monkster which abstracts this for you automatically by implementing a retry mechanism when the replica set is unavailable due to a missing primary or other temporary network error.

Automated Backups

It's a good idea to set up a mechanism to automatically back up your database(s) every day to Amazon S3. If you accidentally delete an entire collection, secondaries will replicate that change and delete it locally as well, so backups will protect you from human error.

Check out mongodb-s3-backup.sh, a shell script I created that will automatically back up one of your databases to S3. You can configure it to run on an arbiter, for example, and have it read the data from a secondary, to avoid impacting the primary's performance. Read the gist for further instructions.

Replica Set Monitoring

It's important to constantly monitor your replica set to avoid downtime or other problematic situations caused by network issues or insufficient resources.

The following should be monitored via a script:

• The health status of the replica set (available via rs.status())
• The health status of each replica set member, from the point of view of each member
• The minimum number of replica set members (should be 3 or more)
• The number of replica set members should be odd, not even
• The existence of a primary replica set member (this may fail if an election is in progress)
• The last heartbeat timestamp from one member to another being less than 3 minutes ago from the point of view of all members
• The oplog date on secondary members, which indicates if they've fallen behind on replication (it should not exceed 15 minutes ago)
• The remaining disk space does not exceed 80% on each and every member
• A recent S3 backup exists in case things go south

I developed a Node.js package that monitors most of these for you called mongomonitor, be sure to check it out!

That's it!

Well done! You've just finished deploying your very own highly-available MongoDB replica set on AWS! Let me know if this helped you!

Is the majority of your AWS monthly bill made up of thousands of on-demand EC2 instance hours, steadily increasing month by month?

If your organization knows it'll be around for years to come, you've probably already purchased Reserved Instances to attempt to lower the cost (by paying in advance for instances which leads to up to 75% cost reduction, depending on how long you pay in advance for). But this pricing model isn't applicable to most startups and small businesses as they may shut down abruptly in a moment's notice. Whether you utilize Reserved Instances or not, there's another nifty trick you can take advantage of to get up to 90% discount on the on-demand EC2 instance price, without paying anything in advance!

Even if your organization doesn't seem to care about optimizing the AWS monthly bill, I'm sure they'd appreciate the huge savings you could harness by utilizing Spot instances correctly! You might even be able to convince the management to organize a company-wide recreational day with the money you save on the monthly AWS bill.

Spot Instances

Their name is a bit misleading -- they are pretty much regular EC2 instances, but simply priced much lower (usually), up to 90% off the on-demand price, and with different life expectancy. Also, they are not a new type of offering -- they actually exist for quite some time now, and yet, most AWS customers still don't take full advantage of them.

Why does AWS offer the same underlying hardware for much less, without any upfront payment? Here's why.

AWS provisions millions of physical servers in advance for their on-demand EC2 instance service. They need to be prepared for when their clients decide they want to spin up 1,000 m4.large instances within a few minutes time, without any notice in advance.

Since AWS will probably always be ready to grant you your on-demand instances when you need them, they have to provision the hardware in advance and keep it running to support your request in a moment's notice. This creates a situation where tons of EC2 compute power is being actively wasted -- physical servers just sitting there doing absolutely nothing, wasting precious resources and money for Amazon (electricity / hardware / cooling / etc), as they wait for clients to utilize them by starting up on-demand instances. There will always be spare compute power in the AWS cloud, waiting for clients to utilize it.

Imagine you had to file in a request for on-demand instances and wait a few hours until AWS manually plugged in and provisioned more physical servers in its data centers to fulfill your request! That wouldn't be much fun.

Spare Compute Power

Spot instances are Amazon's solution to this problem -- it is essentially a stock market for spare compute power. They sell any excess compute power for ridiculously low hourly rates -- usually 80% - 90% off the on-demand price for each supported instance type (not all instance types are available as Spot instances). The Spot price, or the hourly cost of running a Spot instance, is specific to each instance type and availability-zone and is decided by Amazon's secret algorithm which factors in supply and demand of this spare compute power.

Spot instances let you bid on spare Amazon EC2 instances to name your own price for compute capacity. The Spot price fluctuates based on the supply and demand of available EC2 capacity. Your Spot instance is launched when your bid exceeds the current Spot market price, and will continue run until you choose to terminate it, or until the Spot market price exceeds your bid.

Unfortunately, since this is a market, the price can fluctuate both ways -- the Spot price will 95% of the time be substantially less than the on-demand price, however, when demand for compute power within the AWS cloud grows, or when some clients bid too high for Spot instances, the Spot price may spike and actually surpass the on-demand price for a short period of time, shutting down your server within 2 minutes notice if the Spot price exceeds your bid price.

Show Me the Money

So how does the Spot price usually look like?

Here's a graph representing 3 months of Spot pricing history for the m4.large instance type in four different us-east-1 AZs, generated on July 1st of 2016:

For reference, the standard on-demand price for m4.large is $0.12 per instance hour, approximately $86.40 per month.

As you can see from the graph, most of the time, the Spot price fluctuates between $0.0185 (85% less than on-demand, $13.32 / month) and $0.0276 (77% less than on-demand, $19.87 / month), except for when it peaks ridiculously, for relatively short periods of time, most likely due to high demand in a certain AZ. Paying around $15 per month instead of $86.40 for an m4.large instance sounds like a hell of a deal to me!

It's also worth noting that most of the time, the Spot price peaks in only one AZ at a time. So if you spread out your Spot instances in multiple AZs (you should already be doing this for high availability anyway), there is much less chance that all of your Spot instances will terminate at once.

You can view an up-to-date pricing history graph by checking out the Spot instance launch wizard in the AWS Console.

Fear of Spot Instances

I speculate that AWS customers are afraid of using Spot for the following reasons:

• The workflow involved in provisioning them is cluttered and messy
• People are scared off by the fact that Spot instances may shut down abruptly as the Spot price increases past their max bid price
• Spot instances are not supported in all AWS services (for example, in Elastic Beanstalk, however, there are hacks to make them work in EB as well)

Furthermore, most people don't really understand how to utilize Spot instances safely. They tend to go with a radical approach to using Spot -- all or nothing. You see, you shouldn't be relying on Spot instances for 100% of your workload. What you should be doing is supplementing your on-demand instances with Spot instances, where appropriate.

I'd say it's pretty much a safe bet to utilize Spot for 70% of your stateless workload, provided you have at least 10 total instances. Since the Spot price usually won't spike in all AZs at once, you should be good to go. However, the percentage of Spot instances you employ in your workload is definitely application and task-specific. Make your own decision on how much Spot to supplement your scalable environment with.

There is one exception though -- feel free to use 100% Spot instances for any stateless app that is not meant for production -- development and test servers are absolutely fine for Spot, as long as you're OK with suffering a bit of downtime every once in a while if the Spot price peaks.

Bidding

How do you know how much to bid for each Spot instance? A good practice is to simply bid 100% of the on-demand price. That way, you'll never pay more than if the instance was a standard on-demand, and you won't be charged excessively when the Spot price spikes uncontrollably. The best part is that you don't pay your max bid price, but instead, you pay the current Spot price. If you set your max bid price to 100% of the on-demand price, you have absolutely nothing to lose when using Spot!

You can then simply replace terminating Spot instances with on-demands and suffer no blow to your wallet, and hopefully with no noticeable service degradation, provided you supplemented correctly.

Supplementation Example

A backend API service running on 20 on-demand m4.large instances could instead be running on 10 on-demand and 10 Spot instances, setting the maximum bid price at 100% the on-demand price. You could then set up a monitoring service to automatically increase the number of on-demand instances when Spot instances get terminated, and terminate the on-demand when the Spots are back up.

Spot Isn't for Everything

Note that you shouldn't be using Spot for running any sensitive workloads. If it isn't clear enough already, you should NOT run your databases on Spot. Spot instances are meant for workloads which do not persist any sensitive data to a local disk, since they may shut down abruptly as the Spot price increases past your bid price. They're perfect as web servers, API backends, Hadoop, etc. Any kind of workload that can be interrupted and replaced by an on-demand instance without any need for backup and restore. Basically, any stateless application or task that can be interrupted safely.

Spot Blocks

AWS also provides an option to guarantee uptime of up to 6 hours for your Spot instance, in exchange for a higher Spot price. You won't be affected by a price spike, but you'll be paying a bit more for each instance hour. And your instance will always be terminated after 6 hours.

You may prefer this type of Spot instance if you have some one-time task that you need less than 6 hours to finish and don't want it interrupted.

Using Spot

There are several ways to utilize Spot instances:

• Manually request them using the Spot Requests console which is not recommended as there is no resiliency here -- once the Spot price exceeds your bid price, your instances will shut down and AWS will not restore them after the price goes back down
• Request a "Spot Fleet" in the Spot Requests console to attempt to maintain your target capacity by relaunching Spot instances after the price goes down again
• Configure an EC2 Launch Configuration to use Spot instances instead of on-demand and hook up an Auto Scaling Group to use your launch configuration (recommended)
• Configure an Elastic Beanstalk environment to utilize 100% Spot instances (not recommended except for non-production workloads)

I'd recommend going with the Launch Configuration method, as it's the least cluttered. I tend to find the Spot requests console a bit messy to work with.

If you wish to supplement an existing Elastic Load Balancer with Spot instances, simply create another Auto Scaling Group, duplicate its Launch Configuration, modify it to use Spot instances, and finally, attach it to your existing ELB. Your ELB will now forward traffic to both Auto Scaling Groups, and you'll be able to easily increase or decrease the number of desired instances in either Auto Scaling Group to maintain a good on-demand-to-Spot-instances ratio.

That's it! Let me know how you take advantage of Spot instances to save tons of money on your AWS bill in the comments below!

Any tech company that provides its goods and services over the Internet, whether it be in the form of a dashboard interface or feature-rich API, needs to prepare for the inevitable and unexpected hiccups that plague service uptime.

Service Degradation

It manifests itself in various forms, ranging from increased API response times, elevated API error rates, DNS provider outages, datacenter blackouts, database lockups, DDoS attacks, and so on.

An organization can only do so much to avoid service degradation, but sometimes, life just gets in the way. Whether it be a new employee that somehow managed to push untested code to the main production boxes, or even worse, forgetting to renew HTTPS certificates, these things just happen sometimes.

Transparency

The best way for organizations to handle these kinds of situations is with complete transparency. This can easily be achieved by setting up a status page that reflects your service's operational status. You can set up this status page to display uptime metric graphs, any recent incidents, what is being done to resolve them, and an incident history.

There are several paid status page solutions available, but if your budget is tight, luckily, there's Cachet.

Cachet is a beautiful and powerful open source status page system, a free replacement to services such as StatusPage.io, Status.io and others.

Cachet is indeed free and open-source, and one of the greatest things about it is that it's easy to customize its theme to fit your branding.

Here's a demo of Cachet.

The only downside of Cachet is that its installation docs are very lacking. I had to waste nearly 4 hours trying to get it to install successfully. But no matter, I've managed to nail down the perfect installation steps necessary to get it up and running in no time!

Setting Things Up

Let's begin by spinning up a new Ubuntu 14.04 LTS instance. Choose whichever cloud provider you like most, but do consider its reliability history as this status page should be an always-up, go-to place for when your service goes down.

A good idea is to host this instance in a completely different region than the one with all of your production instances, so that in case an entire availability zone or region goes down, the status page shall prevail.

Instance Configuration

As for the instance compute power, you can definitely cut costs here -- I'd even suggest going with a t2.nano if you're using AWS, it costs about $4.50 a month.

Make sure to assign a public IP to the server, as well as allow access on port 80 to all IPs, and on 22 to your IP address.

Finally, set up an A Record in your domain's DNS record management interface so that status.you.com will point to the server you just created.

Install Dependencies

First things first, update the package cache:

sudo apt-get update

Let's install some basic requirements for Cachet, including Apache2 and PHP:

sudo apt-get install git curl apache2 php5 libapache2-mod-php5 php5-gd php5-apcu php5-mcrypt php5-sqlite php5-cli

Next, install Composer, a dependency manager for PHP:

curl -sS https://getcomposer.org/installer | sudo php -- --install-dir=/usr/local/bin --filename=composer

Finally, enable Apache's mod_rewrite as it's required for Cachet to work:

sudo a2enmod rewrite

Install Cachet

Head over to the ubuntu user's home directory:

cd ~

Let's clone Cachet to the server. At the time of writing, the latest stable version of Cachet is v2.2.2.

Check for the latest stable version of Cachet in the official releases page and in case there's a newer version number, plug it into the following command (instead of v2.2.2):

git clone https://github.com/cachethq/Cachet.git
cd Cachet
git checkout v2.2.2

There is actually a dependency reference error in v2.2.2 that Cachet project maintainers have not yet fixed in the stable branch. So let's manually fix it by running the following command:

sed -i 's/use Illuminate\\Support\\Facades\\Str;/use Illuminate\\Support\\Str;/g' app/Http/Controllers/Dashboard/SettingsController.php

Install PHP Dependencies

Install all of the Composer depenencies:

composer install --no-dev -o

Configure Cachet

Create an .env file with the following contents to have Cachet write to a local SQLite database, which is fast and easiest to configure:

nano .env

Paste the following contents inside:

APP_ENV=production
APP_DEBUG=false
APP_URL=http://status.you.com
APP_KEY=

DB_DRIVER=sqlite

CACHE_DRIVER=file
SESSION_DRIVER=file
QUEUE_DRIVER=sync
CACHET_EMOJI=false

MAIL_DRIVER=smtp
MAIL_HOST=null
MAIL_PORT=null
MAIL_USERNAME=null
MAIL_PASSWORD=null
MAIL_ADDRESS=null
MAIL_NAME=null
MAIL_ENCRYPTION=tls

REDIS_HOST=null
REDIS_DATABASE=null
REDIS_PORT=null

GITHUB_TOKEN=null

Note: Make sure to replace status.you.com with the DNS hostname you set up earlier.

Run the following commands to prepare for the installation:

php artisan key:generate
php artisan config:clear
touch ./database/database.sqlite
chmod -R 777 .env storage database bootstrap/cache

Finally, run the following command to install Cachet:

php artisan app:install

If all goes well, great!

If not, check Cachet/storage/logs/laravel-YYYY-MM-DD.log for the stack trace. It's most likely a permission issue.

Now all that's left is to set up Apache to serve Cachet's public/ folder.

Configure Apache

Run the following commands to link /var/www/html to Cachet's public/ directory:

sudo mv /var/www/html /var/www/html-old
sudo ln -s /home/ubuntu/Cachet/public /var/www/html

Let's permit .htaccess directives by editing the Apache2 configuration:

sudo nano /etc/apache2/sites-available/000-default.conf

Find:

DocumentRoot /var/www/html

Add below:

<Directory /var/www/html>
        Options Indexes FollowSymLinks
        AllowOverride All
        Require all granted
</Directory>

Finally, restart Apache2 for changes to take effect:

sudo service apache2 restart

Setup Wizard

Open a web browser and head over to your status page hostname. You should see the following setup screen:

Leave the default cache/session drivers as-is and click Next.

On the next page, you'll be asked to enter some details about the status page, such as its name, URL, time zone, language, etc.

On the third setup page, you'll be asked to set up an administrator account for managing the status page.

When you're done filling everything in, click Go to dashboard. You'll then be presented with this marvelous login screen:

Within the dashboard, you'll be able to add service components (e.g. API / DB / Website / etc), incidents, metrics (graphs), and much more.

To set up HTTPS for your status page, follow this awesome tutorial by DigitalOcean to set up Apache with a free Let's Encrypt certificate.

If you want your customers to be able to subscribe to e-mail alerts, check out the Cachet e-mail setup docs.

Well done, you've successfully set up a status page for your service! Now, it's up to you to set up mechanisms to automatically update this status page with new incidents or updated metric data, using the Cachet API.

There's just no way around it. Whether it's sending your users welcome e-mails, password reset requests, purchase receipts, or billing reminders, almost any web-based service eventually needs to start sending transactional e-mail to its users.

Usually, this is a daunting task, mainly because sending responsive transactional e-mail is actually not so easy to pull off:

• You need to build or import an e-mail template and inject your text into it
• You need to prepare a plaintext version of the e-mail to send along with the HTML e-mail
• You need to inline the CSS in the template (can't use the <style> tag when sending e-mails)
• You need to make sure the template is responsive (as most of the time people will actually be reading it on their mobile device)

And usually this will end up cluttering your JavaScript code with HTML and <br /> statements all over the place, not to mention wasting time that could have been spent building your actual product.

But not anymore.

Mailgen

Mailgen is a Node.js package I built that generates clean, responsive HTML e-mails for you, without having to get down and dirty with the actual templating, inlining, responsiveness, etc.

Programmatically create beautiful e-mails using plain old JavaScript.

Simply pass in your text, and mailgen will do the rest.

This simple code:

var Mailgen = require('mailgen');

// Configure mailgen by setting a theme and your product info
var mailGenerator = new Mailgen({
    theme: 'default',
    product: {
        // Appears in header & footer of e-mails
        name: 'Mailgen',
        link: 'https://mailgen.js/'
        // Optional product logo
        // logo: 'https://mailgen.js/img/logo.png'
    }
});

// Prepare email contents
var email = {
    body: {
        name: 'John Appleseed',
        intro: 'Welcome to Mailgen! We’re very excited to have you on board.',
        action: {
            instructions: 'To get started with Mailgen, please click here:',
            button: {
                color: 'green',
                text: 'Confirm Your Account',
                link: 'https://mailgen.js/confirm?s=d9729feb74992cc3482b350163a1a010'
            }
        },
        outro: 'Need help, or have questions? Just reply to this email, we\'d love to help.'
    }
};

// Generate an HTML email using mailgen
var emailBody = mailGenerator.generate(email);

Generates this awesome e-mail:

You can customize the generated e-mails by entering your product name and logo, as well as providing a custom theme file to be used instead of the default theme. I intend to add at least 5 more built-in themes to mailgen to make things awesome right out of the box.

Interesting enough, two days after I published mailgen, it had already been downloaded 1,000+ times on npm and starred 200+ times on GitHub. It exploded so fast I was caught off guard and had to stop everything I was doing to take care of the feature requests piling up. But hey, who am I to complain, finally one of my open-source projects gets a crazy amount of attention!

What do you think about mailgen? Have any ideas on how to improve it? Let me know in the comments below or by opening a GitHub issue!

npm, a.k.a. the Node Package Manager, is a developer-friendly command-line package manager included with Node.js. It makes it super-easy to install other people's JavaScript packages to extend your projects as well as publish your own JavaScript code with the world.

That exciting feeling you get when you publish an open-source project on GitHub? This one's even better, because npm makes it dead-simple for people to use your package. It's only an npm install away. And if your package does something useful, people will find it without you having to spread the word about it -- developers actively search npmjs.com for packages all the time to instantly add worlds of functionality to their apps.

Due to the magnitude of npm, some critics even go so far as to claim that Node.js developers simply npm install whatever they need their app to do and never write a single line of code themselves, and that's actually not so far-fetched as it sounds, nor is it a bad thing. Why not build upon the experience of other developers that were faced with the same exact task at hand?

An Introduction to npm

Skip this if you're familiar with how npm works.

The npm CLI works by reading and writing to a file called package.json within your project's root which looks similar to this:

{
  "name": "my-cool-package",
  "version": "1.0.0",
  "description": "A cool package for demonstration purposes",
  "main": "index.js",
  "author": "Elad Nava <eladnava@gmail.com>",
  "license": "Apache-2.0",
  "dependencies": {
    "express": "^4.13.4",
    "mongoose": "^4.4.17"
  }
}

Install an npm package by running the following command within your project's root directory:

npm install express --save

npm will then fetch the popular Express package and extract it into the node_modules/ folder within your project's root directory.

The --save argument instructs npm to add the package and its installed version to your project's package.json.

When other people want to work on your project, they simply clone it to their workstation and run:

npm install

All of the dependencies listed within your package.json will then be automatically installed to their local machine.

A Package for Everything

The coolest thing about npm is that there's a package for almost everything. It is, by far, the largest package manager for any development language. Simply search npm for the functionality you're looking to add to your project, and there's a pretty good chance you'll find a package that does exactly what you need!

But sometimes, you'll be working on something so unique that you won't be able to find anything like it on npm. And since you love open source and want to give back to this amazing community, you'll want to publish your awesome package and give it a catchy name that will be easy to remember and to npm install. But how do you go about doing that?

Publishing Your First Package

It's actually easier than it sounds. Let's get to it!

Starting Out

Let's begin by naming your package. You'll want to pick a name that is both self-explanatory and easy to remember. Some of the most popular npm packages tend to go for English words:

• express
• request
• async
• forever
• underscore

And the list goes on... But it's definitely not mandatory to pick an English word. Just make sure that the name corresponds to the functionality of your package, as that will help people find it with ease.

One thing to note here: npm doesn't scope your packages to your account like GitHub scopes your repositories. This means that you should double-check that someone didn't already publish a package with the desired name by plugging it into this URL:
https://www.npmjs.com/package/package-name-goes-here

You gotta love npm for that 404 page. =) If you are presented with a different page, it probably means that the package name is already taken up by someone else.

Functionality

It's time to think about the core functionality of your package. For the sake of this example, let's build a simple package called is-null-or-empty, which will receive a string and return true if it's null, undefined, or empty, or false in all other cases. Feel free to substitute the package name and logic with your own idea.

This is how the package would be used:

var isNullOrEmpty = require('is-null-or-empty');

console.log(isNullOrEmpty("")); // true
console.log(isNullOrEmpty(null)); // true
console.log(isNullOrEmpty(undefined)); // true

console.log(isNullOrEmpty("Hello World")); // false

Pretty simple package, right? That's perfectly fine. Writing tiny packages that do just one thing is a practice known as Hyper Modular JavaScript, and it encourages the use of tiny packages that do one thing well to accomplish a more complex goal, as well as DRYing up our code.

The name I chose for the package is a little long, but you can definitely understand what it's supposed to do when it's named so verbosely. Sometimes our packages do something so specific that it would make sense to name them this way, so that people will find them easily and understand their purpose instantly.

Source Control

Alright, now that we've got both a name and an idea for your package, let's begin to write the actual code.

First things first -- we should set up source control. GitHub is recommended for this as npm integrates nicely with it, as we'll see later.

Create a new repository called is-null-or-empty under your GitHub account by visiting:
https://github.com/new

Next, open up a terminal and run the following to create a directory for your package:

mkdir is-null-or-empty  
cd is-null-or-empty  

Once inside, let's initialize a local git repository for your package, as well as hook up a remote origin to it (the GitHub repo you just created).

git init .  
git remote add origin https://github.com/{your-username}/is-null-or-empty.git  

Make sure to replace {your-username} with your GitHub username.

Package Metadata

Next, let's create a basic package.json file by running the following:

npm init  

npm will now ask us to input some details about the project. You can pretty much press Enter all the way through, but make sure to fill in the following fields:

• Author: First Last <email@provider.com>
• Description: Checks whether a given string is null or empty.
• License : MIT / GPL-3.0 / Apache-2.0 (Check out http://choosealicense.com for help with this)

For the rest of the options, you can leave the default values as they are.

Notice how npm will automatically detect the GitHub repository you configured in the previous step. It will link to your repository in the npm package listing page for people to view the package source and help contribute to your package.

Finally, enter yes to write the package.json file to the disk.

Write the Code

Finally, the part you've been waiting for! Writing your package's oh-so-complicated logic.

We need to create a file called index.js as that is your package's configured entry point (specified using the main property in your package.json). When others require your package, this is the file that will be run first.

Open up your favorite JavaScript IDE (give VS Code a try!) and create a new file called index.js, pasting the following inside it:

// Main package function
function isNullOrEmpty(input) {
    // Returns true if the input is either undefined, null, or empty, false otherwise
    return (input === undefined || input === null || input === '');
}

// Make the main function available to other packages that require us
module.exports = isNullOrEmpty;

Notice that module.exports declaration. We must explicitly tell Node.js what methods we want to make accessible to others by using this declaration. Without it, no one will be able to access the isNullOrEmpty() function we defined inside.

Write an Example File

The best way to demonstrate how to use your package is to write an example script that makes use of it, documenting the return values of your package. Add the following example.js to your project with the following content:

// Change './index' to 'is-null-or-empty' if you use this code outside of this package
var isNullOrEmpty = require('./index');

console.log(isNullOrEmpty("")); // true
console.log(isNullOrEmpty(null)); // true
console.log(isNullOrEmpty(undefined)); // true

console.log(isNullOrEmpty("Hello World")); // false

Notice how the require statement calls for ./index instead of is-null-or-empty. In order to require your own package within your package's code, we have to directly reference its entry point, which is index.js, by writing require('./index'). Anyone else that installs your package will be able to require('is-null-or-empty') as expected.

Test It Out

Now it's time to make sure your package actually works! Run the example.js file with an IDE of your choice or simply by invoking node:

node example.js

As expected, here's the output:

Documentation

This one's actually easier than you think. Simply create a README.md file within your project with the following Markdown-styled content for a basic Node.js package documentation:

# is-null-or-empty

A Node.js package that checks whether a given string is null or empty. A basic package for an npm publish tutorial.

## Usage

First, install the package using npm:

    npm install is-null-or-empty --save

Then, require the package and use it like so:

    var isNullOrEmpty = require('is-null-or-empty');

    console.log(isNullOrEmpty("")); // true
    console.log(isNullOrEmpty(null)); // true
    console.log(isNullOrEmpty(undefined)); // true

    console.log(isNullOrEmpty("Hello World")); // false

## License

Apache 2.0

The README.md is crucial -- almost no one will bother to sift through your package code to attempt to understand how to actually use it, especially if it's a large and complicated package.

Commit everything into your git repo and push it up to GitHub:

git add index.js package.json example.js README.md
git commit -m "Initial commit"
git push origin master

Publish the Package!

And now, the moment you've been waiting for -- publishing your package for the whole world to see!

If this is indeed your first package, you'll need to register on npm by running npm adduser. If you're already registered on npm, use npm login instead.

Here comes the big one:

npm publish

This one might take a few seconds, but once it's done -- so are you!

Congratulations! You just published your first npm package like a boss! Go ahead and run npm install is-null-or-empty and attempt to use your package in another project! =)