Go challenge: validating UK bank account numbers

Standard

As I was reading through the SEPA specification, I found that it was not that simple to check if a UK bank account number was valid or not. If you’re not familiar with UK banks, they don’t use IBAN to transfer money within the UK, but a combination of a sort code and an account number. A sort code identifies the bank’s branch and each account has got an account number. A sort code is a 6 digits number and an account number can be between 6 and 11 digits, but most of them are 8 digits long.

For example, here is a valid UK bank account:

  • Sort code: 107999
  • Account number: 88837491

Algorithms to check if a UK bank account is valid

A very common way to check if a number (bank account, credit card, parking ticket…) is valid, is to apply a modulus algorithm. You perform an operation on each digit (addition, multiplication by a weight, substitution…), when you reach the end you divide by a specific number and you check that the remainder of the division is equal to something. Seems easy, right? Well, this is not that simple for UK bank accounts. In fact, if you want to go through the official specification on the Vocalink website, you will see that they use 2 algorithms, but they have also 15 exceptions to take into account (and some of them are weird or tricky to handle!). You will need to adapt the way you compute the modulus value according to a weight table also.

From the specification to a package

Reading the specification was interesting, but what really motivated me to code a Go package to solve this problem was the fact that test cases where provided in the specification! What a dream: the specification offers you 34 test cases, and they cover nearly all the exceptions. I jumped on the opportunity, it’s not that often that you are offered with a way to check that what you have done is actually right. In fact, I followed a Test Driven Developemnt aproach and it really guided me during the development and especially the refactoring.

Getting the code

The code is available on GitHub under the MIT license and should be well documented and tested. As always, pull requests and bug reports are welcome!

Here is an example:

package main

import (
    "fmt"

    "github.com/AntoineAugusti/moduluschecking/models"
    "github.com/AntoineAugusti/moduluschecking/parsers"
    "github.com/AntoineAugusti/moduluschecking/resolvers"
)

func main() {
    // Read the modulus weight table and the sorting
    // code substitution table from the data folder
    parser := parsers.CreateFileParser()

    // The resolver handles the verification of the validity of
    // bank accounts according to the data obtained by the parser
    resolver := resolvers.NewResolver(parser)

    // This helper method handles special cases for
    // bank accounts from:
    // - National Westminster Bank plc (10 or 11 digits with possible presence of dashes, for account numbers)
    // - Co-Operative Bank plc (10 digits for account numbers)
    // - Santander (9 digits for account numbers)
    // - banks with 6 or 7 digits for account numbers
    bankAccount := models.CreateBankAccount("089999", "66374958")

    // Check if the created bank account is valid against the rules
    fmt.Println(resolver.IsValid(bankAccount))
}

Continuous integration and code coverage in Golang

Standard

It took me some time to find the right setup and the right tools to achieve something not that complicated: continuous integration and coverage reports for Golang projects hosted on GitHub.

I’m happy to share my configuration with you, hopefully it will save you some time. I’m using Travis CI for the continuous integration platform and Codecov for code coverage reports. Both are free and easy to setup: you can get just log in using your GitHub account, you will be up and running in under 5 minutes.

Here is the Travis file (.travis.yml) I use:

language: go
before_install:
  - go get golang.org/x/tools/cmd/vet
  - go get github.com/modocache/gover
  - go get github.com/vendor/package/...
script:
  # Vet examines Go source code and reports suspicious construct
  - go vet github.com/vendor/package...
  # Run the unit tests suite
  - go test -v ./...
  # Collect coverage reports
  - go list -f '{{if len .TestGoFiles}}"go test -coverprofile={{.Dir}}/.coverprofile {{.ImportPath}}"{{end}}' ./... | xargs -i sh -c {}
  # Merge coverage reports
  - gover . coverprofile.txt
after_success:
  # Send coverage reports to Codecov
  - bash < (curl -s https://codecov.io/bash) -f coverprofile.txt

Replace github.com/vendor/package with your GitHub URL and you're good to go! You will be protected against yourself or contributors for your package. Unit tests will not break and coverage will not decrease. Or at least you will know when it happens!

Bonus: fancy badges

I like to put at the beginning of every README file a few information:

  • The status of the latest build (green is reassuring)
  • The software license, so that people immediately know if it's okay to use it for their project
  • A link to the GoDoc website, for documentation
  • The percentage of code covered by unit tests

If you want to do the same, here is what you can write at the very top of your README.md file:

# Travis CI for the master branch
[![Travis CI](https://img.shields.io/travis/vendor/package/master.svg?style=flat-square)](https://travis-ci.org/vendor/package)
# Note that this is for the MIT license and it expects a LICENSE.md file
[![Software License](https://img.shields.io/badge/License-MIT-orange.svg?style=flat-square)](https://github.com/vendor/package/blob/master/LICENSE.md)
# Link to GoDoc
[![GoDoc](https://img.shields.io/badge/godoc-reference-blue.svg?style=flat-square)](https://godoc.org/github.com/vendor/package)
# Codecov for the master branch
[![Coverage Status](http://codecov.io/github/vendor/package/coverage.svg?branch=master)](http://codecov.io/github/vendor/package?branch=master)

One more time, don’t forget to replace vendor/package (even in URLs) with your details and you’re good to go!

Demo

Head to AntoineAugusti/colors to see what it looks like.

Happy coding!

Word segmentation library in Golang

Standard

I’ve been into Golang lately, and today I’m glad to announce my second open source project in Golang, following the feature flags API. My second package is all about word segmentation.

What is the word segmentation problem?

Word segmentation is the process of dividing a phrase without spaces back into its constituent parts. For example, consider a phrase like thisisatest. Humans can immediately identify that the correct phrase should be this is a test. But for machines, this is a tricky problem.

An approach to this problem

A basic idea would be to use a dictionary, and then to try to split words if the current chunk of letters is a valid word. But then you run into issues with sentences like peanutbutter that you will split with this approach as pea nut butter instead of peanut butter.

The idea was to take advantage of frequencies of words in a corpus. This is where the concept of a n-gram is used. In the fields of computational linguistics and probability, an n-gram is a contiguous sequence of n items from a given sequence of text or speech. The items can be phonemes, syllables, letters, words or base pairs according to the application.

For example, this is an extract of some unigrams in a corpus composed of 1,024,908,267,229 words distributed by the Linguistic Data Consortium.

used 421438139
go 421086358
b 419765694
work 419483948
last 417601616
most 416210411
music 414028837
buy 410780176
data 406908328
make 405084642
them 403000411
should 402028056

Using unigrams and bigrams, we can score an arrangement of words. This is what is done in the score method for example.

Concurrency and channels

This was also a great opportunity for me to work with channels, because some parts of the program can be run in parallel. I’m just starting to work around goroutines and channels, but I really like it!

Take a look at the source code and the documentation on GitHub: github.com/AntoineAugusti/wordsegmentation

Feature flags API in golang

Standard

Over the last few months, I’ve been interested in golang (the Go language) but I didn’t know what to build to really try it. Sure, I’ve done the exercises from the online tutorial and I’ve read the awesome website Go by example, but I didn’t have a real use-case yet. Until a few days ago when I decided to build an API related to feature flags!

What are feature flags?

Feature flags let you enable or disable some features of your application, for example when you’re under unexpected traffic or when you want to let some users try a new feature you’ve been working on. They decouple feature release and code deployment, so that you can release features whenever you want, instead of whenever the code happens to ship.

With this package, you can enable the access of a feature for:

  • specific user IDs
  • specific groups
  • a percentage of your user base
  • everyone
  • no one

And you can combine things! You can give access to a feature for users in the group dev or admin and for users 1337 and 42 if you want to.

What I’ve learned

I guess it’s a rather complete project because it involves a storage layer (a key-value store, with bolt), some logic around a simple model (what is a feature? How do we control access to a feature?) and an HTTP layer (with the default HTTP server and gorilla/mux). Moreover I’ve tried to write some tests, and it was really interesting to discover the “Go way” to do it!

Anyway, I’ve learned a lot and I’m fairly happy with the codebase, but if you spot anything that can be improved or that is wrong, please do get in touch with me (GitHub issues and tweets are perfect).

Here is the source code: github.com/AntoineAugusti/feature-flags.

Keeping my brain busy in my free time

Standard

For the last few months, I’ve tried to do sometimes in my free time what is often called “code katas”.

A code kata is an exercise in programming which helps a programmer hone their skills through practice and repetition.

Basically, you try to solve problems by writing algorithms. If you don’t know what to solve, there are a lot of platforms you can look at to find exercises:
CodinGame
Prologin (in French)
Rosalind (bioinformatics challenges)
Kaggle (machine learning competitions)

On most of these platforms, you can “play” alone. You are given a problem with detailed instructions, a dataset and an example of the expected output. Your goal will be to solve this problem. You will know if you have successfully solved the problem when you will submit your solution (usually you can use the language of your choice) thanks to automated unit tests.

I like to solve unusual problems, and most of the time they are more difficult than what I work on on a daily basis (at least when you’ve done enough exercices). This is a great opportunity to discover advanced algorithms or to learn a new language. Because you do this in your time it’s always nice to know than these problems can usually be solved in less than 1 hour, and also that you will not need to maintain your implementation in the future! When you’ve successfully solved a problem, you’re done for real 🙂

If you wanna check what I’ve already done, and maybe to see how I’ve done it, I’ve got a public GitHub repository dedicated to this: github.com/AntoineAugusti/katas.

Decorator pattern and repositories

Standard

My use case

Lately I’ve been using a lot the decorator pattern with repositories on Teen Quotes. My use case is somewhat simple: I use a relational database (MySQL) but sometimes I what to cache the results of some queries in a key-value store (Redis / Memcached for instance). With something like that, I don’t need to hit my database for queries that are always run or are slow to run, I’ll hit my key-value store instead. It’ll reduce the pressure on my database and will give some results faster for the application.

The decorator pattern

If you’re not familiar with the decorator pattern yet, it’s quite simple to use and I’m sure you’ll love it in no time. Basically, the decorator pattern allows behavior to be added to an individual object, either statically or dynamically, without affecting the behavior of other objects from the same class. As Richard Bagshaw said, the idea is that you take an object, and you wrap this object in another object which provides additional functionality, and you keep wrapping extra classes repeatedly for each additional requirement.

If you want to see some real world examples, continue to read this blog post or go directly to Laracasts.

Some code

I’ll show you something I’ve been working on last week: the ability to add tags to quotes. A “tag” is like a category for a “quote” (a post, an article, whatever you want to call it). I’m using Laravel with Eloquent for my relational database. I’ve created an interface called TagRepository.

namespace TeenQuotes\Tags\Repositories;

use TeenQuotes\Tags\Models\Tag;
use TeenQuotes\Quotes\Models\Quote;

class DbTagRepository implements TagRepository {

  /**
   * Create a new tag
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag
   */
  public function create($name)
  {
    return Tag::create(compact('name'));
  }

  /**
   * Get a tag thanks to its name
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag|null
   */
  public function getByName($name)
  {
    return Tag::whereName($name)->first();
  }

  /**
   * Add a tag to a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function tagQuote(Quote $q, Tag $t)
  {
    $q->tags()->attach($t);
  }

  /**
   * Remove a tag from a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function untagQuote(Quote $q, Tag $t)
  {
    $q->tags()->detach($t);
  }

  /**
   * Get a list of tags for a given quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @return array
   */
  public function tagsForQuote(Quote $q)
  {
    return $q->tags()->lists('name');
  }

  /**
   * Get the total number of quotes having a tag
   *
   * @param  \TeenQuotes\Tags\Models\Tag $t
   * @return int
   */
  public function totalQuotesForTag(Tag $t)
  {
    return $t->quotes()->count();
  }
}

Pretty simple stuff, I’m sure you’ve seen this multiple times. Let’s move on to the interesting part: the caching layer. We will create a new class CachingTagRepository implementing the same interface TagRepository. The key thing is that we’ll require a TagRepository class to be given in the constructor of this new class. Ultimately, we will pass the DB layer here.

 
namespace TeenQuotes\Tags\Repositories;

use Cache;
use TeenQuotes\Tags\Models\Tag;
use TeenQuotes\Quotes\Models\Quote;

class CachingTagRepository implements TagRepository {

  /**
   * @var \TeenQuotes\Tags\Repositories\TagRepository
   */
  private $tags;

  public function __construct(TagRepository $tags)
  {
    // The key thing is here: we assume we've already
    // a class that is implementing the interface.
    // We can rely on that!
    $this->tags = $tags;
  }

  /**
   * Create a new tag
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag
   */
  public function create($name)
  {
    return $this->tags->create($name);
  }

  /**
   * Get a tag thanks to its name
   *
   * @param  string $name
   * @return \TeenQuotes\Tags\Models\Tag|null
   */
  public function getByName($name)
  {
    $callback = function() use ($name)
    {
      return $this->tags->getByName($name);
    };

    return Cache::rememberForever('tags.name-'.$name, $callback);
  }

  /**
   * Add a tag to a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function tagQuote(Quote $q, Tag $t)
  {
    Cache::forget($this->cacheNameForListTags($q));

    $keyTotal = $this->cacheNameTotalQuotesForTag($t);

    if (Cache::has($keyTotal))
      Cache::increment($keyTotal);

    return $this->tags->tagQuote($q, $t);
  }

  /**
   * Remove a tag from a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @param  \TeenQuotes\Tags\Models\Tag $t
   */
  public function untagQuote(Quote $q, Tag $t)
  {
    Cache::forget($this->cacheNameForListTags($q));

    $keyTotal = $this->cacheNameTotalQuotesForTag($t);

    if (Cache::has($keyTotal))
      Cache::decrement($keyTotal);

    return $this->tags->untagQuote($q, $t);
  }

  /**
   * Get a list of tags for a given quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @return array
   */
  public function tagsForQuote(Quote $q)
  {
    $key = $this->cacheNameForListTags($q);

    $callback = function() use($q)
    {
      return $this->tags->tagsForQuote($q);
    };

    return Cache::remember($key, 10, $callback);
  }

  /**
   * Get the total number of quotes having a tag
   *
   * @param  \TeenQuotes\Tags\Models\Tag $t
   * @return int
   */
  public function totalQuotesForTag(Tag $t)
  {
    $key = $this->cacheNameTotalQuotesForTag($t);

    $callback = function() use ($t)
    {
      return $this->tags->totalQuotesForTag($t);
    };

    return Cache::remember($key, 10, $callback);
  }

  /**
   * Get the key name when we list tags for a quote
   *
   * @param  \TeenQuotes\Quotes\Models\Quote $q
   * @return string
   */
  private function cacheNameForListTags(Quote $q)
  {
    return 'tags.quote-'.$q->id.'.list-name';
  }

  /**
   * Get the key name to have the number of quotes
   * having a tag
   *
   * @param  \TeenQuotes\Tags\Models\Tag $t
   * @return string
   */
  private function cacheNameTotalQuotesForTag(Tag $t)
  {
    return 'tags.tag-'.$t->name.'.total-quotes';
  }
}

You see, we do some things before (or after) calling the initial implementation, to add some functionalities (here a caching layer). Sometimes we directly defer to the initial implementation (see the create method).

Bonus: registering that in the IoC container

Let’s bind our TagRepository interface to the caching layer and the storage layer in a service provider!

namespace TeenQuotes\Tags;

use Illuminate\Support\ServiceProvider;
use TeenQuotes\Tags\Repositories\CachingTagRepository;
use TeenQuotes\Tags\Repositories\DbTagRepository;
use TeenQuotes\Tags\Repositories\TagRepository;

class TagsServiceProvider extends ServiceProvider {

  /**
   * Bootstrap the application events.
   *
   * @return void
   */
  public function boot()
  {
      //
  }

  /**
   * Register the service provider.
   *
   * @return void
   */
  public function register()
  {
      $this->registerBindings();
  }

  private function registerBindings()
  {
      $this->app->bind(TagRepository::class, function()
      {
          return new CachingTagRepository(new DbTagRepository);
      });
  }
}

Et voilà ! Happy coding!

Why Twitter was down for 5 hours. It’s all about calendars.

Standard

Twitter was down for many users this Monday, between midnight and 5 AM, CET. Android app users were logged out and the API was not working during a few hours. A bug in a line of code caused the service to think that it was 29 December, 2015.

Gregorian VS ISO calendars

The ISO week numbering system uses the YYYY format for the year instead of the Gregorian calendar’s yyyy. It then looks at which week of the year it is, and then uses a date digit with 1 starting on Monday. So, for example, Tuesday of the 50th week of 2014 would have been 2014-W50-2 in ISO week format. The problem comes in when January 1 of the new year ends up falling on a date that doesn’t get along well with the ISO week format. The first day of 2015 will start on a Thursday, whereas the ISO standard expects the first week of the year to start on “the Monday that contains the first Thursday in January.” In 2014-2015, that would be January 1st, 2015. That’s why Twitter believed that we were in 2015 on Monday, because “the first Thursday in January” is in fact this week.

Afterwards, some parts of the system believed we were almost a year later, while other were perfectly aware that we were still in 2014. Users were not able to log in (and were logged out) because tokens have a limited lifetime and therefore are not valid a year later.

Dates are hard to handle

Dates are really hard to handle when programming, and it’s very easy to make a mistake without breaking a thing until the day it will eventually explode everything because calendars or timezones are different. The same kind of mistake happened two years ago with the “Do Not Disturb” feature on iPhones, and it isn’t the last time it will happen for sure.

Using JavaScript / jQuery to detect if someone is using AdBlock

Standard

Let’s face it: ads are not cool. I know, but sometimes you have no other choice than to use them to pay your server / your coffee / whatever. And if you are using ads, you know that some people will not see your ads because they are using a plugin that blocks ads on your website. You want to reach these visitors and just say

“Hey, we know ads are not cool, but please make an exception for us or give us a small amount of money, just to support our work”

On Teen Quotes I am showing a friendly message in place of the only ad to say what I have to say. Here is what people with an ad blocker are seeing:
adblock

But let’s get back to the point: detect visitors that are using an ad blocker with JavaScript.

Some code

Let’s write some code. You’re going to write the most simple JS file you have ever written. Ready? Here it is:

var isNaughtyVisitor = false;

BOOM! That was fast, isn’t it? In fact, we don’t care about the variable name or its value, we just want to define a new variable. But this variable needs to be defined in a file called ads.js that you will include in your HTML. Why so? This is were it gets funny. Ad blockers are a little dumb, so when they see a file named something like ad.js or ads.js, they will block the request. And if the request was blocked… your variable will not be defined!

The second step is just to test in your regular JS file of your application if the variable was defined. This file needs to be include after the previous file, ads.js. You can add something like this:

$(document).ready(function() {
    // The div that will add to the DOM if the visitor is 
    // using an ad blocker
    var div = 'Your amazing HTML block here';
    
    // The script was never called, probably using an ad blocker
    if (typeof(isNaughtyVisitor) == "undefined") {
        // Insert the div wherever you want
        $("#footer").before(div);
        // Send the event to Google Analytics if you want to track
        ga('send', 'event', 'ads', 'hidden');
    // A friendly user!
    } else {
        // Send the event to Google Analytics if you want to track
        ga('send', 'event', 'ads', 'displayed');
    }
});

Pretty darn simple, and really effective. The downside is that friendly users will have to make an extra HTTP request. Yep, I know, it’s not fair for them.

Using ImageMagick to crop images

Standard

ImageMagick is a PHP library for image processing and image generation. It is a requirement for a popular library, stojg/crop that offers various cropping techniques. The library can be installed easily via Composer, but it requires ImageMagick to work.

Installing ImageMagick on Ubuntu / Debian

You are lucky, there is a package for that. It is just a one line command:

sudo apt-get install php5-imagick

It will grab ImageMagick, build the PHP extension and enable it in your php.ini file. You may have to restart your web server or PHP-FPM to reflect the changes.

Using ImageMagick on Travis CI

If you are running a continuous integration test suite, you may be interested to test some methods that are using the ImageMagick library under the hood. In order to do that, the ImageMagick library needs to be installed on the Travis’ VM. At the time, the ImageMagick library is already pulled in for every VM, but the PHP extension is not enabled by default. We will still try to grab the library before enabling it, just to be cautious. You’ll have to edit your travis.yml file and add the following lines to your before_script:

before_script:
  - sudo apt-get update -q
  - sudo apt-get install -y imagemagick
  - printf "\n" | pecl install imagick

The last line is a very dirty hack: when running the command pecl install imagick, you have to hit “Enter” to autodetect some settings. That’s why you need to pipe a new line to this command. Dirty, but it works, you’ll be able to test methods using ImageMagick afterwards.

Fighting against suicide on Teen Quotes

Standard

Teen Quotes lets teenagers share their daily thoughts and feelings. Since people speak their mind, we receive happy quotes about friendship and love, regular life inspiring quotes and sad quotes. And sometimes, unfortunately, really really sad quotes.

At Teen Quotes, we have a very strict publishing policy. Every quote gets reviewed by a human before being published. If we don’t like a quote, we can edit it a little bit to make it publishable or we can refuse it.

How to manage refused quotes

In the past, when we received a quote that we didn’t want to publish, here is what happened:

  • The submitted quote goes through moderation and gets refused. As an immediate result, it will never be published on Teen Quotes.
  • The author of the quote receives an email telling that its quote was refused with a few hints to improve: verify the English, be sure that a similar quote wasn’t published before and try to write an original quote.

This type of email is sent a lot because we refuse around 75 % of the submitted quotes. We would be super happy to explain in every email why this specific quote was refused but it would take too much time during moderation.

But I am convinced that sometimes we should contact the author of the quote, in specific situations. One of the possible situation is described in this post’s title: suicide promotion / suicide thoughts in a quote.

Submitting super sad quotes is a real alarm

As already said, users submit quotes on Teen Quotes to inspire others. Most of our visitors come just to get inspired, to relate to others, to put words on feelings. We deeply think that we shouldn’t publish quotes that promote suicide thoughts. Everyone has ups and downs, but suicide should not be an option. So we protect other members from Teen Quotes by refusing such quotes. And today, we are going one step further.

Trying to help authors that need assistance

In the upcoming major version of Teen Quotes, if a user submits a really sad quote, we will send him a personal email to make sure that everything is right and we will include some useful links if he needs to get help from non-profit organisation around the world. As a software engineer, it is usually difficult to have an impact in someone’s life. Moving a button 2 pixels to the left will not make a huge difference. We deeply hope that we will be able to help people with this feature. It really matters to us.