DDD East Anglia 2013 Conference Write-Up

logo-smallThis past Saturday, 29th June 2013, saw the inaugural DDD East Anglia conference.  This is the latest addition to the DeveloperDeveloperDeveloper events that take place all over the UK and sometimes around the world!  I was there, this being only my second ever DDD event that I’d attended.

DDD East Anglia was set on the grounds of the extensive Cambridge University in a building called “The Hauser Forum”.  For those attendees like myself who were unfamiliar with the area or the university campus, it was a little tricky to find exactly where to go.  The DDDEA team did make a map available on the website, but the designated car park that we were to use was cordoned off when I arrived!  After some driving around the campus, and with the help of a fellow attendee who was in the same situation as myself, I managed to find another car park that could be used.  I must admit that better signposting for both the car parking and the actual Hauser Forum building would have helped tremendously here.  As you can imagine, Cambridge University is not a small place and it’s fairly easy to get lost amongst the myriad of buildings on campus.

The event itself had 3 parallel tracks of sessions, with 5 sessions throughout the day in each track.  As is often the case with most DDD Events, once the agenda was announced, I found myself in the difficult position of having to choose one particular session over another as there were a number of timeslots where multiple sessions that I’d really like to attend were running concurrently.  As annoying as this can sometimes be, it’s testament to quality and diversity of the sessions available at the various DDD events.  East Anglia’s DDD was no different.

As I’d arrived slightly late (due to the car parking shenanigans) I quickly signed in and went along to seminar room 1 where the introduction was taking place.  After a brief round of introductions, we were off and running with the first session.  There had been some last minute changes to the published agenda, so without knowing quite where I wanted to be, I didn’t have to move from Seminar Room 1 and found myself in Dave Sussman’s session entitled, “SignalR: Ready For Real-Time”.

Dave started out by talking about how SignalR is built upon the notions of persistent connections and hubs on the server-side and that hubs are built on top of persistent connections and simply offer a higher level of abstraction.  SignalR, at it’s most basic, is a server-side hub (or persistent connection) through which all messages and data flows and this hub then broadcasts that data back to each connected client.  Other than this, and for the client-side of the equation, Dave tells us that SignalR is effectively one big jQuery module!

Some of the complexities that SignalR wraps up and abstracts away from the developer is the requirement to determine the best communication protocol to use in a given situation.  SignalR uses web sockets as the default method of communication, if the client supports such a protocol.  Web Sockets are a relatively new protocol that provide true duplex communication between client and server.  This facilitates cool functionality such as server-side push to the client, however, if Web Sockets are not available SignalR will seamlessly “downgrade” the connection protocol to HTTP long-polling – which uses a standard HTTP connection that is kept alive for a long time in order to receive the response from the server.

Dave starts to show us a demo, which fails to work the first time.  Dave had planned this however, and proceeded to tell us about one of the most common problems in getting a simple SignalR demo application up and running: Adding the call to .MapHubs (a requirement to register the routes of the Hubs that have been defined on the server-side) after all of the other route registration has been done.  This causes SignalR to fail to generate some dynamic JavaScript code that is required for the client.  The resolution is simply to place the call to .MapHubs before any calls to the other MVC route registrations. 

Dave tells us that SignalR doesn’t have to be in the browser.  We can create many other types of application (Console apps, WinForms Apps etc.) that connect to the HubConnection object over HTTP and then we can create a proxy in the client application that can send and receive the required messages over HTTP to the SignalR hub on the server!  Also, although it’s seen as an ASP.NET addition, SignalR isn’t dependent upon the ASP.NET runtime.  You can use it, via JavaScript, in a simple standalone HTML page with just the inclusion of a few JavaScript files and a bit of your own JavaScript to create and interact with a jQuery $._hubConnection which allows sending and receiving messages and data to the SignalR server-side Hub.

Although SignalR’s most used and default function on the Hub is the ability to broadcast a received message back to all connected clients (as in the ubiquitous “chat” sample application), SignalR has the ability to have the server send messages to only one or more specific clients.  This is done by sending a given message to a specific client based upon that client’s unique ID, which the Hub keeps in an internal static list of connected clients.  Clients can also be “grouped” too if needed so that multiple clients (but not all connected clients) can receive a certain message.  There’s no inherent state in the server-side Hub.  Therefore, things like the server-side collection of connected clients is declared as static to ensure state is maintained.  The Hub class itself is re-instantiated with each message that needs processing.  Methods of the Hub class are Tasks, so clients can send a message to the server hub and be notified sometime later when the task is completed (for example if performing some long running operation such as database backup etc.)

We’re told that there’s no built-in persistence with SignalR, and we should be aware that messages can (and sometimes do!) get lost in transit.  SignalR can, however, be configured to run over a message bus (For example, Windows Azure Message Bus) and this can provide the persistence and improved guarantee of delivery of messages.

Finally, although the “classic” demo application for SignalR is that of a simple “chat” application with simple text being passed back and forth between client and server, SignalR is not restricted to sending text.  You can send any object!  The only requirement here is that the object be able to be serialized for communication over the wire.  The objects that can be passed across the wire are serialized, by default, with JSON (internally using Newtonsoft’s JSON2 library).

After a quick break, during which there was tea, coffee and some lovely Danish pastries available, I was back in the same Seminar Room 1 to attend Mark Rendle’s session entitled, “The densest, fastest-moving talk ever”.  Mark’s session wasn’t really about any single topic, but consisted of Mark writing and developing a simple browser-based TODO List application that utilised quite a number of interesting technologies.  These were Mark’s own Simple.Web and Simple.Data, along with some TypeScript, AngularJS and Bootstrap!

As a result of the very nature of this session, which contained no slides, demos or monologue regarding a specific technology, it was incredibly difficult to take notes on this particular talk.  It was simply Mark, doing his developer thing on a big screen.  All code.  Of course, throughout the session, Mark would talk about what he was doing and give some background as to the what and the why.  As I’m personally unfamiliar with TypeScript and AngularJS it was at times difficult to follow along with what and why Mark was making the choices he did when choosing to utilise one or more of these technologies.  Mark’s usage of his own Simple.Web and Simple.Data frameworks were easier to understand, and although I’ve not used either of these frameworks before, they both looked incredibly useful and lightweight to allow you to get basic database reading & writing within a simple web-based application up and running quite quickly.

After 30 minutes of intense coding including what appeared to be an immense amount of set-up and configuration of AngularJS routing, Mark can show us his application which is displaying his TODO items (from his previously prepared SQL Server database) in a lovely Bootstrap-styled webpage.  We’re only reading data at the moment, with no persistence back to the DB, but Mark spends the next 30 minutes plugging that in and putting it all together (with even more insane AngularJS configuration!).  By the end of the session, we do indeed have a rudimentary TODO List application!

I must admit that I feel I would have got a lot more from this session if I already knew more about the frameworks that Mark was using, specifically AngularJS which appears to be a rather extensive framework that can do everything you’d want to do in client-side JavaScript/HTML when building a web application.  Nonetheless, it was fun and enjoyable to watch Mark pounding out code.  Also, Mark’s inimitable and very humorous style of delivery made this session a whirlwind of information but really fun to attend.

Another break followed after Mark’s session with more tea, coffee and a smorgasbord of chocolate-based snacks positioned conveniently on tables just outside of each seminar room (more on the food later!).  Once the break was over, it was time for the final session of the morning and before the lunch break.  This one was Rob Ashton’s “Outside-In Testing of MVC”.

Rob’s opening gambit in this session is to tell us that his talk isn’t really about testing MVC, but its about testing general web applications and will include some MVC within it.  A slight bait and switch, and clearly Rob’s style of humour.  He’s mostly a Ruby developer these days so there’s little wonder there’s only a small amount of MVC within the session!  That said, the general tone of the talk is to explore ways of testing web applications from the outermost layer – the User Interface – and how to achieve that in a way that‘s fast, scalable and non-brittle.  To that extent, it doesn’t really matter what language the web application under test is written in!

Rob talks about TDD and that often people trying to get started in TDD often get it wrong.  This is very similar to what Ian Cooper talked in his “TDD, where did it all go wrong?” talk that he’s given recently in a number of places.  I attended Ian’s talk at a recent local user group.  Rob says he doesn’t focus so much on “traditional” TDD, and that often having complete tests that start at the UI layer and test a discrete piece of functionality in a complete end-to-end way is very often the best of all testing worlds.  Of course, the real key to being able to do this is to keep those tests fast.

Rob says he’s specifically avoiding definitions in his talk.  He just wants to talk about what he does and how it really helps him when he does these very things.  To demonstrate, he starts with the example of starting a brand new project.  He tells us that if we’re working in brownfield application development, we may as well give up all hope!  :(

Rob says that we start with a test.  This is a BDD style test, and follows the standard “Given, When, Then” format.  Rob uses CoffeeScript to write his tests as its strict handling of white-space forces him to keep his tests short, to the point, and easily readable, but we can use any language we like for the tests, even C#.

Rob says he’s fairly dismissive of the current set of tools often used for running BDD tests, such as Cucumber.  He says it can add a lot of noise to the test script that’s really unnecessary and often causes the test script wording to become more detached and abstracted from what it is the test should actually be doing in relation to the application itself.  So we are asked the question, What do we need to run our test?” – Merely a web browser and a web server!

In order to keep the tests fast, we must use a “headless” browser.  These are implementations of browser functionality but without the actual UI and Chrome of a real web browser.  One such headless browser is PhantomJS.  Using such a tool allows us to run a test that hits our webpage, performs some behaviour – adds text to a textbox, clicks a button etc. – and verifies the result of those actions, all from the command line.  Rob is quick to suggest that we shouldn’t use PhantomJS directly as then our tests will be tightly coupled to the framework we’re running them within.  Rob suggests using WebDriver (part of the Selenium suite of web browser automation tools) in conjunction with PhantomJS as that will provide a level of abstraction, and thereby not coupling the tests tightly to the browser (or headless browser) being used.  This level of abstraction is what allows the actual test scripts themselves to be written in any language of our choosing.  It just needs to be a language that can communicate with the WebDriver API.

Rob then proceeds to show us a demo of running multiple UI tests in a console window.  These tests are loading a real webpage, interacting with that page – often involving submission of data to the server to be persisted in someway, and asserting that some action has happened as a result of that interaction.  They’re testing the complete end-to-end process.  The first thing to note is that these tests are fast, very fast!.  Rob is spitting out some simple diagnostic timings with each test, and each test is completing in approx. 500ms!

Rob goes on to suggest ways of ensuring that, when we write our tests, they’re not brittle and too closely tied to the specific ID’s or layout of the page elements within the page that we’re testing.  He mentions one of the best tools to come from the Ruby world, Capybara.  Rob says that there’s a .NET version of Capybara called Coypu although it’s not quite as feature-complete as Capybara.  Both of these tools aim to allow intelligent automation of the browser testing process and help make tests readable, robust, fast to write with less duplication and less tightly coupled to the UI.  They help to prevent brittle tests that are heavily bound to UI elements.  For example, the tools try multiple ways to fill in a “username” textbox when instructed by first looking for the specific ID, but then intelligently looking for a <label for=”username”> if the ID is not found and using the textbox associated with the label.  If that’s not found, the tool will then intelligently try to find a textbox that happens to be “near” to where some static text saying “Username” may be on the page.

Rob suggests not to bother with “fast” unit tests.  He says to make your UI tests faster!  You’ll run them more frequently, and a fast UI test means a fast UI.  If we achieve this, we’re not just testing the functionality, but by ensuring we have a suite of UI tests that run very fast, we will by virtue of that, have an actual application and UI that runs very fast.  This is a win-win situation!

Rob proceeds to build up a demo application that we can use to run some tests against.  He does this to show us that he’s not going to concern himself with databases or persistence at this point – he’s only storing things in an in-memory collection.  Decisions about persistence and storage should come at the very end of the development as by then, we’ll have a lot more information about what that persistence layer needs to be (i.e.. a document database, SQL Server for more complex queries etc.)  This helps to keep the UI tests fast!

Rob then proceeds to give us a few tips on MVC-specific development, and also about how to compose our unit tests when we have to step-down to that level.  He says that our Controllers should be very lightweight and that we shouldn’t bother testing them.  You’ve got UI tests that cover that anyway.  He states that, “If your controller has more than one IF statement, then it shouldn’t!”.  Controllers should be performing the minimal amount of work.  Rob says that if a certain part of the UI or page (say a register/signup form) has complex validation logic, we should test that validation in isolation in it’s own test(s).   Rob says that ActionFilters are bad.  They’re hard to test properly (usually needing horrible mocking of HTTPContext etc.) and they often hide complexity and business logic.  This logic is better placed in the model.  We should also endeavour have our unit-level tests not touch any part of the MVC framework.  If we do need to do that, have a helper method that abstracts that away and allows the test code to not directly touch MVC at all.

To close, Rob gives us the “key takeaways” from his talk:  Follow your nose, focus on the pain and keep the feedback loop fast.  Slides for Rob’s talk are available here.

20130629_124147After Rob’s talk, it was time for lunch.  This was provided by the DDDEA team, and consisted of sandwiches, crisps, a drink and even more chocolate-based confectionary.  There was also the token gesture of a piece of fruit, I suppose to give the impression that there was some healthy items within there!

There was even the ability to sit outside of the main room in the Hauser Forum and eat lunch in an al-fresco style.  It was a beautiful day, so many attendees did just that.  The view from the tables on this balcony was lovely.

 

20130629_124107As is often the case at events such as these, there were a number of informal “grok” talks that took place over the lunchtime period.  These are usually 10 minutes talks from any member of the audience that cares to get up and talk about a subject that interests them or that they’re passionate about. 

Since I was so busy stuffing my face with the lovely lunch that was so kindly provided, I had missed the first of the grok talks.  I managed to miss most of the second grok talk, too, which was given by Dan Maharry about his experiences writing technical books.  As I only caught the very end of Dan’s talk, I saw only one slide upon which was the wise words, "Copy editors are good. Ghost-writers are bad."  Dan did conclude that whilst writing a technical manual can be very challenging at times it is worth it when, three months after completing your book, you receive a large package from the publishers with 20-30 copies of your book in there with your own name in print!

The last grok talk, which I did catch, was given by Richard Duttton on life as a Software Team Lead at Red Bull Racing.  Richard  spoke about the team itself and what kind of software they produce to help the Formula 1 team build a better and faster car.  Richard answered the question of “What’s it like to work in F1?”.  He said there’s long hours and high pressure but it’s great to travel and see how the software you write affects the cars and the race first hand.

Red Bull Racing development team is about 40 people strong.  About half of these have a MATLAB background rather than .NET/C#.  Richard’s main role is developing software for data distribution and analysis.  He writes software that is used on the race pit walls as well as being used back at HQ.  They can get a data reading from the car back to the IT systems in the HQ within 4 seconds from anywhere in the world!  The main data they capture is GPS data, Telemetry data and Timing data.  Within each of these categories of data, there can be 1000’s of individual data points that are captured.

Richard spoke about the team’s development methodology and said that they do “sort-of” agile, but not true agile.  It’s short sprints that align with the F1 race calendar.  There are debriefs after each race.  When 2 races are back-to-back on consecutive weekends, there’s only around 6 hours of development time between these two races!

The main software languages used are C#.NET 4.5 (VS2010 & VS2012 with TFS and centralized build system) but they mostly develop WPF applications (with some legacy WinForms stuff in there as well).  There’s also a lot of MATLAB.  They still have to support Windows XP as an OS as well as more modern platforms like Windows Phone 8.

After the lunch time and grok talk sessions were over, it was back to the scheduled agenda.  There were two more sessions left for the day, and my first talk of the afternoon was Ashic Mahtab’s “Why use DDD, CQRS and Event Sourcing?

Ashic starts by ensuring that everyone is familiar with the terminology of DDD, CQRS and Event Sourcing.  He gives us the 10 second elevator pitch of what each acronym/technology is to ensure we know.  He says that he’s not going to go into detail of what these things are, but rather why and when you should use them.  For the record, DDD is Domain-Driven Design, CQRS is Command Query Responsibility Segregation and is about having two different models for reading vs. writing of data, and Event Sourcing is about not writing a lot of changes to a database record in one go, but to write the different stages of changes to the record over time.  The current state of the record is then derived by combining all the changes (like delta differences).

He says that very often applications and systems are designed as “one big model”.  This usually doesn’t work out so well in the end.  Ashic talks about the traditional layered top-down N-Tier architecture and suggests that this is a bad model to follow these days.  Going through many layers makes no sense, and this is demonstrated especially well when looking at reading vs. writing data – something directly addressed by CQRS.  Having your code go through a layer that (for example) enforces referential integrity when only needing to read data rather than writing it is unnecessary as referential integrity can never be violated when reading data, only when data is being written.

Ashic continues his talk by discussing the notion of a ubiquitous language and that very often, a ubiquitous language isn’t ubiquitous.  Different people calls things by different names.  This is often manifested between disparate areas of an enterprise.  The business analysts may call something by one name, whilst the IT staff may call the same thing by a different name.    We need to use a ubiquitous language, but we also need to understand that it’s often only ubiquitous within a “bounded context”.  A bounded context is a ring-fenced area where a single “language” can be used by everyone within that area of the enterprise and is ubiquitous within that context.  This provides a “delimited applicability of a particular model, gives team members a clear and shared understanding of what has to be consistent and what can develop independently.”.

Ashic goes on to talk about the choices of the specific technology stack and how those choices can impact many areas of a project.  A single technology stack, such as the common WISA Microsoft-based stack for web applications (the Windows/Microsoft equivalent of the even more common LAMP stack) can often reduce IT expenditure within an enterprise, but those cost savings can be mitigated by the complexity of developing part of a complete system using a technology that’s not an ideal fit for the purpose.  An example may be using SQL Server to store documents or binary data, when a document –oriented database would be a much more appropriate solution.

Ashic tells a tale of his current client who only have one big model for their solution that comprises of around 238 individual projects in a single Visual Studio solution.  A simple feature change that was only 3 lines of code required the entire solution to be redeployed.  This in turn required testing/QA, compliance verification and other related disciplines to be re-performed across the entire solution even though only a tiny portion had actually changed.  The “one big model” had forced them into this situation, whereas multiple, separate models communicating between each other by passing messages in a service-oriented approach would have facilitated a much smaller footprint for deployment, and thus a smaller sized application that needed testing and verification.

Ashic tells us that event sourcing over a RESTful API is a good thing.  Although there’s the possibility of the client application dealing with slightly stale data, it’ll never be “wrong” data as the messages passed over this architecture are immutable.  Also, if you’re using event sourcing, there’s no need to concern yourself about auditing and logging, all individual changes are effectively audited anyway by virtue of the event sourcing mechanism!  Ashic advises caution when applying event sourcing and consideration should be given to where not to apply it.  If all you’re doing in a certain piece of functionality is retrieving a simple list of countries from a database table that perhaps contains only one or two columns, it’s overkill and will only cause you further headaches if applied.

He states that versioning of data is difficult in a relational database model.  You can achieve rudimentary versioning with a version column on your database tables, or a separate audit table, however this is rarely the best approach or delivers the best performance of design.  Event sourcing can significantly help in this regard, too, as rather than versioning an entire record (say a “customer” record which may consist of dozens of fields), you’re versioning a very small and specific amount of data (perhaps only a single field of the customer record).  The event sourcing message that communicates the change in the one field (or a small number of fields) effectively becomes the version itself as multiple changes to that field(s) will be sent over several different immutable messages.

The talk continues with an examination of many of the various tools and technologies that we use today. Dependency Injection, Object mapping (with Automapper for example) and Aspect-Oriented programming.  Ashic ponders whether these things are really good practice.  Not so much the techniques that (for example) a dependency-injection container performs, but whether we need the container itself at all.  He says that before DI Containers came along, we simply used the factory pattern and wrote our own classes to perform the very same functionality.  Perhaps where such techniques can be written by ourselves, we should avoid leaning upon third-party libraries.  After all, dependency injection can often be accomplished in as little as 15 lines of code!  For something a little more complicated such as aspect-oriented programming, Ashic uses the decorator pattern instead.  It’s tried and trusted, and doesn’t re-write the IL of your compiled binaries – something which makes debugging very difficult - like many AoP frameworks do.

Ashic concludes his talk by restating the general theme.  Don’t use “one big model” to design your system.  Create bounded contexts, each with their own model and use a service-oriented architecture to pass messages between these model “islands”.  The major drawback to using this approach to the design of your system is that there’s a fair amount of modelling work to do upfront to ensure that you properly map the domain and can correctly break that down into multiple discreet models that make sense.

20130629_150120After Ashic’s talk, there was the first afternoon break.  Each of the three seminar rooms shuffled out into the main hall area to be greeted with a lovely surprise.  There was a table full of delicious local cheeses, pork pies, grapes and artisan bread, lovingly laid on by Rachel Hawley and the generous folks at Gibraltar software (Thanks guys! I think you can safely say that the spread went down very well with the attendees!)

So after we had all graciously stuffed our faces with the marvellous ploughman’s platter, and wet our whistles with more tea and coffee, it was time for the final session of the day.

The final session for me was Tomas Petricek’s “F# Domain-Specific Languages”.

Tomas starts out by mentioning that F# is a growing language with a growing community across the world - user groups, open source projects etc.  It’s also increasingly being used in a wide variety of companies across many different areas. Credit Suisse use F# for financial processing (perhaps the most obvious market for F#) but the language is also used by companies like Kaggle for machine learning and also companies like GameSys for developing the server-side components used within such software as Facebook games.

Tomas then demos a sample 3D Domain-specific language (or DSL) that composes multiple 3d cylinders, cones and blocks to composite ever more elaborate structures from the component parts.  He shows building a 3D “castle” structure using these parts that combines multiple functions from the domain-specific language, underpinned by F# functions.  He shows that the syntax of the DSL contains very little F# code, only requiring a small number of F# constructs when we come to combine the functions to create larger, more complex functions.

After this demo, Tomas moves on to show us how a DSL for European Call and Put stock options may look.  He explains what call and put options are (Call options are an agreement to buy something at a specific price in the future and Put options are an agreement to sell something at a specific price in the future) and he then shows some F# that wraps functions that model these two options.

Whilst writing this code, Tomas reminds us that everything in F# is statically typed.  Also that everything is immutable.  He talks about how we would proceed to define a domain-specific language for any domain that we may wish to model and create a language for.  He says that we should always start by examining the data that we’ll be working with in the domain.  It’s important to identify the primitives that we’ll be using.  In the case of Tomas’ stock option DSL, his primitives are the call and put options.  It’s from these two primitive functions that further, more complex functions can be created by simply combining these functions in certain ways.  The call and put functions calculate the possible gains and/or losses for each of the two options (call or put) based upon a specific current actual price. Tomas is then able to “pipeline” the data that is output from these functions into a “plot” function to generate a graph that allows us to visualize the data.  He then composes a new function which effectively “merges” the two existing functions before pipelining the result again to create another graph that shows the combined result set on a single graph.  From this we can visualize data points that represent the best possible price at which to either buy or sell our options.

Tomas tells us that F# is great for prototyping as you’re not constrained by any systems or frameworks, you can simply write your functions that accept simple primitive input data, process that data, then output the result.  Further functions are then simply  composed of those more basic functions, and this allows for very quick testing of a given hypotheses or theory.

For some sample F# code, Tomas models the domain first like so:

type Option =
| EuropeanPut of decimal
| EuropeanCall of decimal
| Combine of Option * Option

This is simply modelling the domain and the business language used within that domain.  The actual functionality to implement this business language is defined later.

Tomas then decides that the definition can actually be rewritten to something equivalent but slightly better like so:

type OptionKind = Put | Call

type Option =
| European of OptionKind * decimal
| Combine of Option * Option

He can then combine these two put/call options/functions like so:

let Strangle name lowPrice highPrice = 
    Combine
       (  European(Put, name, lowPrice),
          European(Call, name, highPrice)  )

Strangle is the name of a specific type of option in the real world of stock options and this option is a combination of call and put options that are combined in a very specific way.  The function called Strangle is now defined and is representative of the domain within which it is used.  This makes it a perfect part of the domain-specific language.

Tomas eventually moves onto briefly showing us a DSL for pattern detection.  He shows a plotted graph that can go up and down across the x axis and how we can used F#-defined DSL-specific functions to detect that movement up or down.  We start by defining the “primitives”.  That could be the amount of the movement (say, expressed in pixels or some other arbitrary unit we decide to use), and then a “classifier”.  The classifier tells us in which direction the movement is (i.e. up or down).  With these primitives defined, we can create functions that detect this movement based upon a certain amount of points that are plotted on our graph.  Although Tomas didn’t have time to write the code for this as we watched (we were fairly deep into the talk at this point with only a few minutes left), he showed the code he had prepared earlier running live on the monitor in front of us.  He showed how he could create multiple DSL functions, all derived from the primitives, that could determine trends of the movement of the plotted graph over time.  These included detection of:  Movement of the graph upwards, Movement of the graph downwards and even Movement of the graph in a specific bell curve-style (i.e. a small downwards movement, immediately followed by an upwards movement).  For each of these combined functions, Tomas was able to apply them to the graph in real-time, by simply “wiring up” the graph output - itself a DSL function, in this case a recursive one that simply returned itself with new data with every invocation – to the detection functions of the DSL.

At this point, Tomas was out of time, however what we had just seen was an incredibly impressive display of the expressiveness, the terseness, and the power of F# and how domain-specific languages created using F# can be both very feature rich and functional (pardon the pun!) with the minimum of code.

At this point, the conference was almost over.  We all left our final sessions and re-gathered in the main hall area to finish off the lovely cheese (amazingly there was still some left over from earlier on!) and wait whilst the conference organisers and Hauser Forum staff rearranged the seminar rooms into one big room in which the closing talk from the conference organisers would be given.

After a few minutes, we all shuffled inside the large room, and listened as the DDD organisers thanked the conference sponsors, the speakers, the university staff and finally the attendees for the making the conference the great success that it was. And it was indeed a splendid event.  There were then some prizes and various items of swag to be given away (I didn’t win anything :( ) but I’d had a fantastic day at a very well organised and well run event and I’d learned a lot, too.  Thanks to everyone involved in DDDEA, and I hope it’s just as good next year!

DDD North 2012 Conference Write-up

This past weekend, on Saturday 13th October 2012, the 2nd Developer Developer Developer North conference was held at the University of Bradford.  I attended the conference, which was my first Developer Developer Developer (DDD) event ever, and it was a cracker!

DeveloperDeveloperDeveloper events are a series of conferences held around the UK and in some locations abroad focused primarily at Microsoft/.NET Developers.  The conferences are free to attend and are made possible by the support of a wonderful set of sponsors.

The University Of Bradford was a great venue for the conference.  It had plenty of space and rooms available to accommodate the DDDNorth event which had 5 parallel tracks of talks and 5 sessions throughout the entire day.  Each session was an hour in length with 15 minute breaks in-between the 2 morning sessions and the 3 afternoon sessions.  There was a catered lunch provided free of charge to attendees during the generous 1.5 hour lunch time.

As there were 5 parallel tracks of sessions, it was often a difficult choice to pick just one session to attend.  This was especially true for myself within the 2nd session time-slot, where I really wanted to attend all 5 parallel talks!  Unfortunately, I had to only pick one.

The first talk I attended was Garry Shutler's "10 Practices that make me the developer I am today".  This was a talk aimed at more "entry level" developers, but I thought I'd attend to see if there were a few nuggets of wisdom that I perhaps didn't know.

Garry told us that standards matter, although what they are, doesn't.  Using StyleCop to help enforce standards across your team can help to keep consistency and there's even integration into ReSharper to help with this.  Garry also tells us of the importance of Code Reviews, although much of their value comes when they're done at a "story" (as in, an Agile User-Story) level rather than at a more granular level.  We should learn constantly as no-one else cares about our own personal learning (for both our current and future jobs) and we shouldn't wait for our employers to do this for us.  We should learn new languages, especially ones that are significantly different from those that we use every day.  It's a big investment, but worth it as concepts and paradigms in one language can help us understand similar concepts in other languages.  To help with our learning, we should leverage experience of other developers around us.  It's only obvious once you know!  Testing and Automation are a huge help in getting a faster development feedback loop and allow us effectively "go faster" in our development by correcting our course more frequently.  Within our code, we should trust no-one by ensuring we always implement preconditions to prevent such annoyances as pesky "Null Reference Exceptions" when we were expecting an object to be passed to us, and we should also log excessively, not just errors and exceptions, but everything.  This is a big help when trying to debug issues on production environments where a real debugger can't be used.

The next talk of the day was Liam Westley's "Async C# 5.0 - Patterns for Real World Use" which was a great talk about the Asynchronous programming features introduced into C# 5.0 with the async and await keywords.  Liam's talk specifically focused on the WhenAny and WhenAll methods that perform functionality when working with sets of tasks, and used a concept of copying or downloading music files of varying formats to demonstrate the versatility of the various Async methods.

Liam tells us that we should return a Task instead of a void, where we would have previously done so.  This gives the caller information about what's happened (or happening) during our method.  He then goes on to tell us about the use of the WhenAll method for dealing with lists (i.e. List<Task<string>>) of tasks.  Further processing can happen when all tasks have completed, as all tasks are important here.  Next up, Liam tells us about the very useful and versatile WhenAny method which can be used in numerous ways.  One of which was maintaining a limited batch of a specific number of files when copying/downloading large numbers of files.  The WhenAny method detects completions, removes them from the batch and replaces them with a new file copy task, thereby effectively throttling the downloads to certain batch size.  WhenAny was also used to show it's usefulness in redundancy.  This can be used for competing services where the first to return wins. For example, downloading multiple versions/formats of the same music file, and automatically playing the one that's downloaded first. Other files will continue to download in the background, but can be cancelled if required, needing only a single Cancellation Token for all outstanding tasks.  This technique can also be used for early bailout when we want to cancel outstanding tasks based upon notifications from one or more completed tasks.  The final interesting usage of WhenAny was for interleaving. Liam's demo here showed music files and their associated md5 hashes being downloaded.  Once both the music file and the associated .md5 hash file had downloaded, a hash check could be computed on the file pair.  Very clever stuff indeed!  Liam ended his talk with a mention of a Microsoft white-paper that he recommended us all to download for further reading, "Task-Based Asynchronous Pattern" by Stephen Toub.  Oh, and a rickroll.

After a short break, the following talk was Gemma Cameron's "BDD - Look Ma! No frameworks".  Gemma promised an interactive session with this one, all geared around Behaviour-Driven Development (BDD) and design, and we weren't disappointed.  A very interesting talk that caused all the attendees of the talk to have a real good think about how we would document the behaviour of buying an apple!

Gemma started by asking us to think about why we test. Testing creates good code because it creates a good design for our code.  When we test first, we're forced into making our production code testable.  We should beware of retro-fitting tests to existing code that perhaps was not written using a test-first approach.  This has the potential to "bake-in" bad code by fitting a (passing) unit test around it!  Gemma goes on to say that BDD has often been called "TDD Done Right" and in many ways this is true.  BDD is about us as developers asking "Why?" rather than asking "How?". As developers, we're good at solving problems and thinking about how we might implement a solution, but it's very easy for us to lose sight of why a feature is implemented when viewed from a business requirement perspective.  BDD helps, and if done correctly, forces us to consider the why of a feature's business requirements as that is baked right into our BDD tests forcing us to informatively document those business requirements within our tests.  Adding such an expressiveness to our test code, and specifically an expressiveness that is written in a language understandable not just by developers but by business analysts, project managers and product owners as well will result in bringing developers closer to product owners and including all the other roles in between.

BDD isn't really about unit tests, it's about working from the top-down instead of from the bottom-up. This means we start with the abstract feature requirements (the why?) and gradually drill down into the specifics (the how?) of how we'll implement that within our code.  A popular starting point for organising our BDD tests is to use the GWT (Given When Then) syntax, for example: Given [initial context], when [event occurs], then [ensure some outcomes]. Gemma does point out that although GWT can be a useful starting point, it's often restrictive as it doesn't always lend itself to best expressing our requirements.

Gemma's session continued with all attendees attempting to put this into practice by writing a test that correctly and sufficiently expresses our requirements around purchasing an apple!  Our first collective attempt by the attendees, which followed a GWT syntax was something along the lines of:

public void BuyAnApple()
{
   GivenAppleCosts50();
   AndThatIHaveOneAppleInMyBasket();
   WhenICheckout();
   ThenTotalIs50();
}

Note our use of the GWT syntax for expressing our requirements.  After some further discussion and reflection, we eventually arrived at a much better test:

public void ShopKeeperSellsAnApple()
{
   AppleCosts50();
   CustomerHasAppleInTheirBasket();
   WhenICheckout();
   ThenTotalIs50();
}

Note that here we’ve dispensed with the formal GWT syntax, instead preferring a more natural language to express our requirements and actual real-world behaviour.

In reference to the title of her talk, Gemma was quite down on the use of frameworks to help with the process of writing BDD tests, preferring instead to "hand-code" all of the test syntax.  Frameworks can be helpful, but there's the potential to focus (or lean) too much on the tool or framework rather than getting the behaviour and requirements documented in the common language.  It's this language that we need to work on improving, as this will act as our product documentation both for ourselves as developers and for the business people.  It's this language that's invaluable in letting us know what and why we did something when we return to that code in 6 months time!

After this we had a very nice lunch with sandwiches, fruit and chocolate, all provided by the generous sponsors of the event.  There were a number of Grok Talks during lunch.  These are short 10-15 minute ad-hoc talks given by various attendees of the event.  Unfortunately, I didn't get to attend any of the Grok Talks as I was far too busy stuffing my face!  :)

The first session after lunch was Rob Ashton's "Javascript Sucks And It Doesn't Matter!".  Rob's talk was advertised as being controversial, and he didn't disappoint!  Pretty much straight out of the blocks Rob tells that he doesn't use semi-colons in his JavaScript code, something that he's sure would really annoy Douglas Crockford, the man who's trying to get us all to write more "correct" JavaScript.  Rob goes on to say it's specifically because it'd get up Douglas Crockford's nose that he doesn't use semi-colons!

Rob tells us that JavaScript is a "broken" language in that it's dynamic, allows much flexibility with it's syntax (witness Rob eschewing semi-colons), and will happily perform the strangest type-coercion in order to allow you to add apples to oranges.  Rob goes on to say that, despite JavaScript's fast-and-loose approach, this doesn't really matter as we've got some very nice tools at our disposal to help "keep JavaScript sane".  JSLint & JSHint are both very useful code-quality tools that allow us to have our JavaScript inspected for potential problems including unsafe comparisons, accidentally declaring global variables (rather than locally-scoped ones), un-strict code and stylistic correctness amongst other things.  We can even run these tools from the command line as part of a continuous integration process with the use of node.js, which is effectively JavaScript for the server.

Rob says that our JavaScript code should be tested just as much as we would test our C# code, and to this end, a tool called Zombie comes in very handy.  Zombie is a "headless browser" which is itself written in JavaScript and allows us to put our JavaScript code through it's paces from a command line driven, continuous testing process.  Rob says that we should try to avoid automating a real browser, as we would do with Selenium, as this is a much slower process.  Testing with Zombie is fast and provides that quick feedback loop that we get with a speedy continuous testing process allowing a more rapid develop/debug cycle.

Of course, it's not all about leaning on the tools to prevent bad JavaScript code from being written.  Rob highlights the fact that we need good discipline to write good code.  At this point, Rob also wanted to address the thorny issue of TypeScript.  TypeScript is a new language from Microsoft that is a superset of JavaScript.  It's advertised as "application-scale" JavaScript, compiles to native plain-old JavaScript, and attempts to add some static typing and better object-oriented support to the JavaScript language.  Rob suggests that TypeScript is largely "application-scale" Marketing rather than anything else and points out that much of what TypeScript gives us can be achieved without the need for a new "language".

We all know that JavaScript files and functions can be included within other JavaScript files, simply by referencing them, however, this quickly gets unwieldy and unmaintainable.  Moreover, the ordering of how JavaScript files are loaded is often very important.  This reminds me of my own struggles with developing in "Classic" ASP/VBScript many years ago and just how easy it can be to tie yourself up in knots when including one file within another (which usually, in turn, includes further files) and managing that complex chain of inclusion.  Tools such as RequireJS can help to bring order to this inclusion madness and attempt to make JavaScript more modular by making it easier to ensure the relevant functionality has been loaded and brought into scope before attempting to execute code that will rely upon it.  Other tools such as CommonJS offer similar functionality but also go further by offering additional functionality in the areas of Modules, Packages and Promises (which greatly simplify asynchronous programming).

A simple example of how RequireJS helps to achieve it's goal is seen here:

var otherFile = require("./otherFile.js");
otherFile.Dosomething();

A similar concept to this is available within TypeScript, too:

import otherFile = module(“otherFile”);

This simplification and improvement of managing required/included files is slated to be included within ECMAScript Version 6, currently code-named "Harmony".  ECMAScript is the standardized version of the JavaScript language, ratified by ECMA (European Computer Manufacturers Association), however, as to when Version 6 will be finally released is anyone's guess!

The last session of the day was Ian Cooper’s “Event-Driven Architecture”.  This was a fairly heavyweight talk from Ian that gave us a deep-dive into the concepts and best-practices around architecting an application in an event-driven (or service oriented) approach.

Ian started by reminding us of the 4 tenets of service orientation:

  • Boundaries are explicit
  • Services are autonomous
  • Services share schema and contract, not class
  • Service compatibility is determined based on policy

Event-driven, or service oriented architecture is a set of design principles that, much like object-orientation, help us to architect an application that is composed of individual services.  Services are autonomous “mini-applications” that perform some discreet function.  The entire application is composed of many of these services that will talk to each other via message passing.  There are explicit boundaries between these services and each service passes all of the data required to the next service in the chain in order for that service to perform its function.  Services are effectively “black-boxes” that share nothing of their internal workings or state, and assert their requirements, constraints and capabilities via a public schema or contract.

Ian goes on to discuss the various types of inter-service communication that is available with Service Oriented Architecture.   The simplest type is “Request-Reply”.  This is something we’re all familiar with as it’s exactly how the world-wide web works.  We (the client) request something.  We wait whilst the server composes and sends back to us it’s reply.  A slightly better approach that avoid the necessity for the client to “wait” for the response is something known as “Request-Reaction”.  Here, the requestor (client) no longer has to wait for the data to be sent back.  After an initial acknowledgement of the request by the server, the client can receive the data a later point in time.  Here, the requestor usually “polls” for the result, but can be informed when the result is available (i.e. similar to a callback function).

The next type of communication is “Inversion of communication”.  This allows a main service to push all of it’s events or messages to an external central message queue.  Consumers who are interested in those events can then simply “listen” to that queue.  This helps to reduce the need for systems or services to “know” about each other, thereby reducing coupling between autonomous services even further.  This is helpful as part of the overall architecture as the more one system needs to “know” about another one, the harder it is to integrate those systems into a cohesive whole.  In this “event publishing” scenario, the publisher of the event doesn’t need to know anything about the consumers of those events!  Ian used an example of a hotel’s internal systems with a main “reservation” system publishing a “Reservation Made!” event or message to the external message queue, allowing a “Room Cleaning” service to subsequently view that message and arrange for room service personnel to clean the room pending the hotel guest’s arrival, by watching the central message queue for “Reservation Made!” events.

Ian continued by talking by Messages and Events.  What is an event?  Well, an event is simply a message.  Messages are the data that passes into and out of our services, in a format or schema defined by the service, and allows the service to perform functions based upon that data.  Messages can be either thin or fat (see below for further information) and are usually communicated along a non-durable channel (which essentially means the messages are not persisted to disk) for high throughput.  Messages are passed along Channels and Queues.  Channels allow the passing of messages between services and usually operate in a real-time manner.  Some channels can act as Queues, and queues will often persist messages (thus acting as a durable channel) allowing long-term storage and delaying of messages.  A central message queue would most likely take this approach to it’s handling of messages.  Crucially, channels should only operate on one type of message, using separate channels when different messages types need to be passed around.

Often, channels and queues will operate in conjunction with related services such as a routing service or a transformation service.  Routing services will ensure messages are routed to the correct destination – often determined by a business process or workflow (see orchestration details below), whilst transformation services will ensure messages can be converted (transformed) from one type to another.  This can often involve adding additional data to the message, removing extraneous data from the message that’s no longer required, or it could simply mean changing the schema of the message from one schema to another.

Ian then talked about the concept of Reference Data which is the data used by the services themselves.  This data can be both private data, which is used by the service itself internally, or it can be public data, which is the data that services will pass around within their messages.  Often the public data can be delivered in two different ways.  Ian talked about thin messages and fat messages.  A thin message doesn’t include all of the data that subsequent services may require from the originating service in order to do their jobs.  They are given a small amount of data (keeping the message “thin”) and told where they can go to retrieve further data that they may require.  This usually involves going back to the originating service with a request for that additional data.  On the other hand, messages can be “fat” and include all of the possible data that subsequent services may require in order for them to perform their functions.  They may even have more data available to them than they need.  There are pros and cons to both messages types.  Thin messages create a need to services to perform extra communication in order to retrieve additional data and this in turn requires services to “know more” about each other.  Fat messages avoid this extra communication but create bigger messages and may introduce security issues by exposing so much data, publically, that may not be required by the downstream services that will operate on the messages.

Ian continued by talking about Sagas.  Sagas are like long-running transactions that pass through multiple services.  Ian was keen to point out that actual transactions should never cross service boundaries. This would defeat the concept of services being autonomous and having explicit boundaries.  It’s quite possible, though, that business processes and workflows  will indeed be composed of several discreet steps performed in a specific order, where each of these steps has its functionality provided by a separate service.  The alternative approach to a service-spanning transaction is to raise additional messages (such as a “Reservation Failure” event) that interested consumers must listen for an respond to appropriately.

When many different services are required as part of a long-running process, we utilise an orchestration service to manage the “flow” of the messages.  Orchestration will ensure that the correct events or messages are passed to the appropriate service at the appropriate time, and helps to define the steps of a business process.  An Orchestration service effectively know all about the various services involved in a long-running “story” so that they (the services themselves) don’t have to!

Ian had warned us at the beginning that there was a lot of information to cover in his talk, and at this point we had, unfortunately, run out of time.  It was time for all of the attendees to gather in the main hall for the grand prize draws.  We’d each been given a raffle ticket earlier in the day, and now was the time that we would see if we’d won one of the many, many prizes being given away at the end of the day.  Unfortunately, I didn’t win a thing – although the raffle tickets with numbers either side of mine were called out!  This didn’t matter, though, as I’d had a brilliant day at a very well run event and listened to some impressive speakers talking about incredibly interesting subjects.

Overall, I really enjoyed my first DDD event and I can’t wait until next year to be able to attend DDD North (and hopefully some of the other DDD events around the UK) again!