Using Microsoft Graph API from a daemon process (in Perl on Linux!)

At work, I've been asked to look at how easy (or difficult) it will be to give folk using Microsoft Office365 access to the student timetables.  We already provide a feed into our Google Apps for Education accounts for the students, and this has been very popular.  However The Powers That Be have had a funny five minutes and decided to move future students to Office365 (no, none of us can work out why either!).

Now in the past we've bodged up a means of staff seeing time table information in Office365 (as they've been stuck in there for some years now) using iCal files.  This works but:

• requires the user to make an active decision to register and then past the link into their Office365 Web Access (OWA) calendar setup
• means that Microsoft's servers whack our servers ever 4 hours to update this iCal link (which is OK for a small subset of the staff, but would be less fun for our servers when 15000+ students start to hit them).

So I've been looking at the exciting new Microsoft Graph API that they released a week or two back. Actually its been lurking in beta for a while, but v1.0 appeared more or less as soon as we started to look at how to do this, which is supposed to be the first general release for production use.  According to some Microsoft folk we talked to, this API is going to be the way of the future, so its what we started to look at using (as opposed to the older SOAP based Exchange Web Services API).

The Graph API uses RESTful calls, JSON, and OAuth2.0, so it looks pretty sane.  That was a surprise for me: I'm used to Microsoft stuff looking awful from the start from the point of view of a Linux hacker.  Much of the documentation for the API's OAuth2.0 flows assumes that you're going to be writing a web delivered app.  In this case, the app interacts with the user to get them to log into Azure Active Directory and then delegate rights to do specified things as them to your code.  That isn't much use for a daemon process which is what we want, but luckily Microsoft also implement the "client credentials" flow in OAuth2.0.  This means that you can get a client token (aka client_id) for your daemon application and you can then use the API to swap the client token for a bearer access token that can access any user's data in your Office365 tenancy, limited to the scopes set by the admins. One initial stumbling block for me was that I'm not normally an AD admin in our tenancy, and it looks like you need to be in order to assign application authorization scopes to the app in the Azure management console (luckily our AD guys were OK with giving me admin access to a test tenancy where I could break things to my hearts content whilst working out how this all works).  Still, this is very similar to Google Apps, where you give access scopes to a service account that can then act as other users.

Just to make things a bit sicker, my end of the Graph API calls is coming from Perl scripts sitting on a Linux box.  I'm a Perl hacker, and I've already written Perl modules to wrap up some of the Google APIs in the past, so this isn't overly concerning to me.  Indeed one of the selling points of the Microsoft Graph API's RESTful, OData standards basis is that its pretty much language and platform agnostic.  Its just as happy talking to a Perl script on a Linux box as it is a C# program on a Windows server.

Being version 1.0, the current Microsoft Graph API has a few oddities. I'm not sure why these didn't get fixed in the beta period - maybe the fact that it was a beta put off normal Microsoft developers from using it (use Open Source folk are used to using alpha and beta releases in production, as we've got the code to fix things if they go wrong!).

For example, lets say you want to use Graph to add a member to a Unified Group.  Unified Groups which are an exciting new type of group that can have calendars, files and conversations associated with them (they have nothing to do with local AD groups, existing security or mail enabled groups, calendar groups or distribution groups.  Microsoft really need to stop using the word "group" for new collections of things!). That's easy: there's a documented RESTful call for adding members.  Simiilarly, you can list members of the group. Great - those all work a treat. Now how do you remove members from the group?  Ah.  There doesn't seem to be an API call for that.  Or at least if there is, its not currently documented or its not in the same place as the creating/list members calls. I've flagged it up on Stackoverflow, so hopefully someone will either point me in the right direction or fix the API/documentation. That would be a bit of a show stopper for us though - students are flighty, jittery types who do tend to jump around the modules they are studying so we need to be able to add and remove them from the groups easily (and preferably without them getting an email every time this happens).

On the flip side, Microsoft have said that Unified Groups have some interesting new features, such as being able to have calendars attached directly to them. That sounds like just want we need: we can have a Unified Group for each module's timetable, add the students (and staff teaching it) to this group (possibly by adding the existing local AD module groups into the Azure AD and then adding those groups as members of the Unified Groups) and then fill the group with calendar events for all the lectures, labs, seminars and tutorials. Unfortunately this doesn't seem to work at the moment... the API documentation seems to indicate it should, but I and others are getting errors that the Unified Groups don't have a mailbox.  The odd thing is that I can use the Graph API to add events to an individual user's calendar and if I set the attendees to be the group, it does appear in the group calendar in OWA.  That behaviour is... odd.  But then it could be because I'm not getting how Microsoft intend Office365 calendars to work  I'm used to Google's calendars - ACLs in Google calendars land actually seem to work fine for sharing calendars, although you do have to make quite a few API calls for large numbers of students when setting them up (which Microsoft's Unified Groups would do away with if it worked).

Anyway, I'll keep plugging away at it.  I just hope I don't accidentally turn into the department's Microsoft Graph API "expert".  That would be embarrassing for a Linux hacker!

UPDATE: According to the Marek Rycharski from Microsoft on Stackoverflow, it turns out that the Graph API can't (yet) handle calendars on Unified Groups.  Its "on the roadmap" but no immediate plans for implementing it in the near future.  Drat!  Still at least I got told how to remove users from the Unified Groups ready for when I do need it at some point in the distant future.

Goodreads, Perl and Net::OAuth::Simple

Part of my day job is developing and gluing together library systems.  This week I've been making a start on doing some of this "gluing" by prototyping some code that will hopefully link our LORLS reading list management system with the Goodreads social book reading site.  Now most of our LORLS code is written in either Perl or JavaScript; I tend to write the back end Perl stuff that talks to our databases and my partner in crime Jason Cooper writes the delightful, user friendly front ends in JavaScript.  This means that I needed to get a way for a Perl CGI script to take some ISBNs and then use them to populate a shelf in Goodreads. The first prototype doesn't have to look pretty - indeed my code may well end up being a LORLS API call that does the heavy lifting for some nice pretty JavaScript that Jason is far better at producing than I am!

Luckily, Goodreads has a really well thought out API, so I lunged straight in. They use OAuth 1.0 to authenticate requests to some of the API calls (mostly the ones concerned with updating data, which is exactly what I was up to) so I started looking for a Perl OAuth 1.0 module on CPAN. There's some choice out there! OAuth 1.0 has been round the block for a while so it appears that multiple authors have had a go at making supporting libraries with varying amounts of success and complexity.

So in the spirit of being super helpful, I thought I'd share with you the prototype code that I knocked up today.  Its far, far, far from production ready and there's probably loads of security holes that you'll need to plug.  However it does demonstrate how to do OAuth 1.0 using the Net::OAuth::Simple Perl module and how to do both GET and POST style (view and update) Goodreads API calls.  Its also a great way for me to remember what the heck I did when I next need to use OAuth calls!

First off we have a new Perl module I called Goodreads.pm. Its a super class of the Net::OAuth::Simple module that sets things up to talk to Goodreads and provides a few convenience functions. Its obviously massively stolen from the example in the Net::OAuth::Simple perldoc that comes with the module.

#!/usr/bin/perl

package Goodreads;

use strict;
use base qw(Net::OAuth::Simple);

sub new {
    my $class  = shift;
    my %tokens = @_;

    return $class->SUPER::new( tokens => \%tokens,
                               protocol_version => '1.0',
                               return_undef_on_error => 1,
                               urls   => {
                                   authorization_url => 'http://www.goodreads.com/oauth/authorize',
                                   request_token_url => 'http://www.goodreads.com/oauth/request_token',
                                   access_token_url  => 'http://www.goodreads.com/oauth/access_token',
                               });
}

sub view_restricted_resource {
    my $self = shift;
    my $url  = shift;
    return $self->make_restricted_request($url, 'GET');
}

sub update_restricted_resource {
    my $self = shift;
    my $url          = shift;
    my %extra_params = @_;
    return $self->make_restricted_request($url, 'POST', %extra_params);
}

sub make_restricted_request {
    my $self = shift;
    croak $Net::OAuth::Simple::UNAUTHORIZED unless $self->authorized;

    my( $url, $method, %extras ) = @_;

    my $uri = URI->new( $url );
    my %query = $uri->query_form;
    $uri->query_form( {} );

    $method = lc $method;

    my $content_body = delete $extras{ContentBody};
    my $content_type = delete $extras{ContentType};

    my $request = Net::OAuth::ProtectedResourceRequest->new(
        consumer_key     => $self->consumer_key,
        consumer_secret  => $self->consumer_secret,
        request_url      => $uri,
        request_method   => uc( $method ),
        signature_method => $self->signature_method,
        protocol_version => $self->oauth_1_0a ?
                                   Net::OAuth::PROTOCOL_VERSION_1_0A :
                                   Net::OAuth::PROTOCOL_VERSION_1_0,
        timestamp        => time,
        nonce            => $self->_nonce,
        token            => $self->access_token,
        token_secret     => $self->access_token_secret,
        extra_params     => { %query, %extras },
        );
    $request->sign;
    die "COULDN'T VERIFY! Check OAuth parameters.\n"
        unless $request->verify;

    my $request_url = URI->new( $url );

    my $req = HTTP::Request->new(uc($method) => $request_url);
    $req->header('Authorization' => $request->to_authorization_header);
    if ($content_body) {
        $req->content_type($content_type);
        $req->content_length(length $content_body);
        $req->content($content_body);
    }

    my $response = $self->{browser}->request($req);
    return $response;
}
1;

Next we have the actual CGI script that makes use of this module. This shows how to call the Goodreads.pm (and thus Net::OAuth::Simple) and then do the Goodreads API calls:

#!/usr/bin/perl

use strict;
use CGI;
use CGI::Cookie;
use Goodreads;
use XML::Mini::Document;
use Data::Dumper;

my %tokens;
$tokens{'consumer_key'} =  'YOUR_CONSUMER_KEY_GOES_IN_HERE';
$tokens{'consumer_secret'} = 'YOUR_CONSUMER_SECRET_GOES_IN_HERE';

my $q = new CGI;
my %cookies = fetch CGI::Cookie;

if($cookies{'at'}) {
    $tokens{'access_token'} = $cookies{'at'}->value;
}
if($cookies{'ats'}) {
    $tokens{'access_token_secret'} = $cookies{'ats'}->value;
}

if($q->param('isbns')) {
    $cookies{'isbns'} = $q->param('isbns');
}


my $oauth_token = undef;
if($q->param('authorize') == 1 && $q->param('oauth_token')) {
    $oauth_token = $q->param('oauth_token');
} elsif(defined $q->param('authorize') && !$q->param('authorize')) {
    print $q->header, 
    $q->start_html,
    $q->h1('Not authorized to use Goodreads'),
    $q->p('This user does not allow us to use Goodreads');
    $q->end_html;
    exit;
}

my $app = Goodreads->new(%tokens);

unless ($app->consumer_key && $app->consumer_secret) {
    die "You must go get a consumer key and secret from App\n";
}       

if ($oauth_token) {
    if(!$app->authorized) {
        GetOAuthAccessTokens();
    }
    StartInjection();
} else {
    my $url = $app->get_authorization_url(callback => 'https://example.com/cgi-bin/good-reads/inject');
    my @cookies;
    foreach my $name (qw(request_token request_token_secret)) {
        my $cookie = $q->cookie(-name => $name, -value => $app->$name);
        push @cookies, $cookie;
    }
    push @cookies, $q->cookie(-name => 'isbns',
                              -value => $cookies{'isbns'} || '');
#    print $q->redirect($url);
    print $q->header(-cookie => \@cookies,
                     -status=>'302 Moved',
                     -location=>$url,
                     );
}

exit;

sub GetOAuthAccessTokens {
    foreach my $name (qw(request_token request_token_secret)) {
        my $value = $q->cookie($name);
        $app->$name($value);
    }
    ($tokens{'access_token'}, 
     $tokens{'access_token_secret'}) = 
         $app->request_access_token(
                                    callback => 'https://example.com/cgi-bin/goodreads-inject',
                                    );
}

sub StartInjection {
    my $at_cookie = new CGI::Cookie(-name=>'at',
                                    -value => $tokens{'access_token'});
    my $ats_cookie = new CGI::Cookie(-name => 'ats',
                                     -value => $tokens{'access_token_secret'}
                                     );
    my $isbns_cookie = new CGI::Cookie(-name => 'isbns',
                                       -value => '');
    print $q->header(-cookie=>[$at_cookie,$ats_cookie,$isbns_cookie]);
    print $q->start_html;

    my $user_id = GetUserId();
    if($user_id) {
        my $shelf_id = LoughboroughShelf(user_id => $user_id);
        if($shelf_id) {
            my $isbns = $cookies{'isbns'}->value;
            print $q->p("Got ISBNs list of $isbns");
            AddBooksToShelf(shelf_id => $shelf_id,
                            isbns => $isbns,
                            )
        }
    }
        
    print $q->end_html;
}

sub GetUserId {
    my $user_id = 0;
    my $response = $app->view_restricted_resource(
                                                  'https://www.goodreads.com/api/auth_user'
                                                  );
    if($response->content) {
        my $xml = XML::Mini::Document->new();
        $xml->parse($response->content);
        my $user_xml = $xml->toHash();
        $user_id = $user_xml->{'GoodreadsResponse'}->{'user'}->{'id'};
    }
    return $user_id;
}

sub LoughboroughShelf {
    my $params;
    %{$params} = @_;

    my $shelf_id = 0;
    my $user_id = $params->{'user_id'} || return $shelf_id;
    
    my $response = $app->view_restricted_resource('https://www.goodreads.com/shelf/list.xml?key=' . $tokens{'consumer_key'} . '&user_id=' . $user_id);
    if($response->content) {
        my $xml = XML::Mini::Document->new();
        $xml->parse($response->content);
        my $shelf_xml = $xml->toHash();
        foreach my $this_shelf (@{$shelf_xml->{'GoodreadsResponse'}->{'shelves'}->{'user_shelf'}}) {
            if($this_shelf->{'name'} eq 'loughborough-wishlist') {
                $shelf_id = $this_shelf->{'id'}->{'-content'};
                last;
            }
        }
        if(!$shelf_id) {
            $shelf_id = MakeLoughboroughShelf(user_id => $user_id);
        }
    }
    print $q->p("Returning shelf id of $shelf_id");
    return $shelf_id;
}

sub MakeLoughboroughShelf {
    my $params;
    %{$params} = @_;

    my $shelf_id = 0;
    my $user_id = $params->{'user_id'} || return $shelf_id;

    my $response = $app->update_restricted_resource('https://www.goodreads.com/user_shelves.xml?user_shelf[name]=loughborough-wishlist',
                                               );
    if($response->content) {
        my $xml = XML::Mini::Document->new();
        $xml->parse($response->content);
        my $shelf_xml = $xml->toHash();
        $shelf_id = $shelf_xml->{'user_shelf'}->{'id'}->{'-content'};
        print $q->p("Shelf hash: ".Dumper($shelf_xml));
    }
    return $shelf_id;
}

sub AddBooksToShelf {
    my $params;
    %{$params} = @_;

    my $shelf_id = $params->{'shelf_id'} || return;
    my $isbns = $params->{'isbns'} || return;
    foreach my $isbn (split(',',$isbns)) {
        my $response = $app->view_restricted_resource('https://www.goodreads.com/book/isbn_to_id?key=' . $tokens{'consumer_key'} . '&isbn=' . $isbn);
        if($response->content) {
            my $book_id = $response->content;
            print $q->p("Adding book ID for ISBN $isbn is $book_id");
            $response = $app->update_restricted_resource('http://www.goodreads.com/shelf/add_to_shelf.xml?name=loughborough-wishlist&book_id='.$book_id);
        }
    }
}

You'll obviously need to get a developer consumer key and secret from the Goodreads site and pop them into the variables at the start of the script (no, I'm not sharing mine with you!). The real work is done by the StartInjection() subroutine and the subordinate subroutines that it then calls once the OAuth process has been completed. By this point we've got an access token and its associated secret so we can act as whichever user has allowed us to connect to Goodreads as them. The code will find this user's Goodreads ID, see if they have a bookshelf called "loughborough-wishlist" (and create it if they don't) and then add any books that Goodreads knows about with the given ISBN(s). You'd call this CGI script with a URL something like:

https://example.com/cgi-bin/goodreads-inject?isbns=9781565928282

Anyway, there's a "works for me" simple example of talking to Goodreads from Perl using OAuth 1.0. There's plenty of development work left in turning this into production level code (it needs to be made more secure for a start off, and the access tokens and secret could be cached in a file or database for reuse in subsequent sessions) but I hope some folk find this useful.