The ScrumBan Jira board

This is the part 1 of 4 of the “Lessons learnt from building Data Science systems at Barclays” series.

Agile board

Let’s start with one of the core tool of the agile workflow. We use a Jira board for tracking and organizing all of our projects. We developed a custom board which uses the sprints concept of Scrum but in a more flexible way as in Kanban.



The Scrumban board is configured as following:

  • Horizontally divided in swimlanes (top-down in order of priority):
    • Critical / Blockers
    • Current work
    • Stories backlog
    • Sub-tickets backlog
    • Completed
  • The columns are:
    • To do
    • In progress
    • In review
    • Done / resolved
    • You can optionally have “Ready to release”
  • Quick filters should at least have one filter for each member of the team filtering on its own assigned tickets.

The idea is that during the planning you select from the backlog which high level stories you want to deliver by the end of the sprint (typically 2 weeks long) and then you create subtasks as-you-need.
Reason is that in data science you don’t know what you are about to implement beforehand. Thus you need to investigate-implement-test all the time and as you do it, you discover what to do next. Important is that whatever subtasks is created it is done by the end of the sprint so that the story is completed.

Define stories with a clear goal and a small scope. They should not span over multiple sprints and since that they come with the uncertainty of what tasks will be required, you really need to break a big problem into smaller well-defined problems that are accomplishable no-matter-what.

Avoid having tasks for exploratory analysis or for adding unit tests. Each task should bring some value, potentially a new feature. Each task will then require an exploratory analysis as well as some development and testing. Those steps are already part of the definition of “Done”. See below sections for more explanations about tests and exploratory analysis.

Plan always less than your capabilities. Delivering your stories a few days earlier is a very good sign. Delaying them is bad. If you manage to get your work done by Thursday, spend the whole day of Friday in a pub celebrating your amazing delivery.

In Jira, you must assign each story to one individual but remember that in an agile team either the whole team succeeds or fails. If that person does not manage to finish his tasks on time, it is a team failure. That’s what you have the morning standup for, to make sure everything is under control and team resources are allocated in a way that the sprint is going to be successful.

Never change the scope of your sprints or add tasks that were not planned, unless are required hotfixes. If you are asked to do something else then invite the product owners to join your next sprint planning and only then you can allocate resources for them.
Remember the goal of a sprint is to have a working, even if simplistic, deliverable not solving sparse tasks.

At the end of the sprint have a retrospective meeting to discuss what went well and what not. Make sure to take actions in order to avoid that blockers may appear again in future.


Documentation should be as simple as possible.

  • Releases notes, a page where you can note the major changes since previous version, the list of new tickets that have been merged  (linking to Jira) and a link to a more detailed report.
  • The detailed report contains snapshots of the most recent logs, results, observations, limitations, assumptions and performances of the model/etl/application. Often it contains some charts that can quickly explaining how good the product is. We can use those detailed but concise reports to track how the product is evolving. The release detailed report also contains the help messages of how to run the application and all of the command line interface (CLI) options.
    If all of your tests and procedures are fully automated then this page is simply a copy and paste of the results.
  • The usage of a particular job class or a script with the list of CLI arguments and default values is also accessible using –help argument, many libraries helps you doing that (bash getops, scala Scallop…).
  • Other pages are used to explain the complex part of the logic. Try to reduce those pages only when the logic is very complicated and hard to understand by just reading the code.

Documentation is hard to keep in sync that’s way we want to document what’s new since the last release rather than going through the whole wiki and updating every single page.

Ideally the documentation comes from the source code, unit tests and jira tickets. Individual analysis, findings and insights can be documented separately but they should represent static reports rather than project documentation.

In the hierarchical structure of the pages, we limit the maximum depth to 2. Which means we have the root-level pages with at most one level of children pages. Nested structures make it very hard to find contents when you need them.

Branching and versioning

Code should always and only exist in a git repository. Sparse snippets or random script files should be avoided.

We follow the gitflow branching model where each ticket is mapped as features branch. If you integrate Jira with Stash then from the ticket web page you can automatically create the corresponding branch in the repository using develop as branch base.

You do not need to use the complete gitflow branching model but at least the master, develop and features branches. It’s up to the deployment strategy defining how to handle hotfixes, bugfixes and releases branches. Make sure this strategy is clearly defined and is consistently enforced. See deployment.

Story tickets generally don’t have a branch associated, their sub-tasks have.

Install a git hook that every commit will include as prefix the ticket code (that you can parse out from the branch name). Tracking each commit with the corresponding ticket is a life-saver when in future you will try to reverse engineer what a method is doing and why has been created in first place. Then you can access the whole git history and access the corresponding tickets that touched that piece of code.


Discussions of specific tasks should go into the corresponding jira ticket web page. This will make the conversation public, tracked and anyone can jump into the discussion with the full context available. Also reference files or supporting documents should be attached to the jira ticket itself or in the wiki if they serve as a general purpose. Remember each jira ticket can be linked from the releases wiki page, that means we never lose track of them. Moreover the query engine is quite good.

We found emails to be the worst place for discussions to happen, especially for sharing files that will become soon out-of-date.

When someone sends you an Excel file, reply saying that your laptop does not have an Office installation on it. If you are sharing small data files, tsv or json is way to go.
Avoid comma separated files with quotes wrapping text fields. You want to make your file editable using simple bash commands rather than loading into a csv parsing library.

We tried also mounted shared drives, but confluence is a much better collaborative way to share and organize files with an integrated version control and metadata.

Avoid meetings as much as you can, invent some excuse, ask for a clear agenda beforehand. Educate your colleagues to communicate with you by raising issues. Leave meetings only for important discussions and spend your meeting time for presenting and checkpointing with your stakeholders more frequently.



About Gianmario

Responsible Data Scientist with proven experience of building machine learning products and leadership skills. Co-author of the book "Python Deep Learning", contributor to the “Professional Manifesto for Data Science” and founder of the Data Science Milan community.
This entry was posted in Agile, Big Data and tagged , , , , , . Bookmark the permalink.

5 Responses to The ScrumBan Jira board

  1. Pingback: Lessons learnt from building data-driven production systems at Barclays | Vademecum of Practical Data Science

  2. Ricardo Fonseca says:


    As students of Red River College, we are conducting a study to show that Scrumban can really help an incident based team.

    To support that, we’re running a quick questionnaire as follows (it should take less than 10 minutes):

    If you fill the questionnaire and the e-mail field, we’ll send you the results and the paper at the end of the research.

    Please feel free to participate and share this as we believe that this agile process will help incident based teams in the future.

    Thanks, best regards!


  3. Curtis Reed says:

    I believe you are operating under some misconceptions.
    “In Jira, you must assign each story to one individual …” I don’t believe that is a required field in Jira, so the tool does not force this behavior. The behavior itself lends itself to anti-patterns that the team should avoid. Team ownership of stories is the mindset we want to promote so they are swarmed and completed more quickly. This also reduces WIP and helps stop the “partially done” antipattern we often see. Therefore, when you say “if that person doesn’t complete his tasks it is a team failure” is factually incorrect. If the team doesn’t complete the story, it is an opportunity to ask how they can adapt their process to get better. One or two incomplete stories does NOT mean “team failure”. If they completed 50 points but had one 2-point story not complete, why is that a failure? A 96% success rate is very good. A 50% success rate…not so much. If you are assigning stories to individuals and so you have a lot of WIP and a lot of carry over, then truly it is a team failure to understand the Kanban behaviors that Scrumban is encouraging.
    “Never change the scope of your sprints or add tasks that were not planned” False. Scrum allows change in scope of the sprint, as long as it does not put the Sprint Goals in jeopardy, and if it doesn’t cause significant disruption to the forecast and committed work to be done. Kanban allows change in priority at any time, but unless it is an Expedited new story, we don’t stop what we are doing to work on it. Instead, it jumps to the top of the team queue to be handled next. New tasks (or subtasks, in Jira) can be added as-needed at any time, that is the team’s call. If it changes things significantly, this should be discussed in the Retro to learn what to do better next time.
    The whole branching and versioning section has nothing to do with Scrum or Kanban, but is a development team decision. There are other code management tools besides git, so you’re making an assumption that the team is using only Atlasian tools, when they may be a .net shop using Jira for team work planning and another tool for code management.
    “Avoid meetings as much as you can…”? Does this include the agile meetings, planning meetings, retrospectives, etc.? This is a very curious statement. Planning, coordinating, removing impediments, discussing how to improve, are all central to agile, so suggesting they be avoided is just silly.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.