Coding | DEZERAECOX

Learning to code: it's as easy and as complex as that.

Sun, 05 Jul 2020 15:12:56 +0000

Learning to code: it’s as easy and as complex as that.

Many life scientists have used their time under pandemic-induced lockdown to learn their first programming language. Yay! Coding has many many advantages (reproducibility, more complex analyses, reusability, time-saving) that I have espoused all too many times before.

Many people start with a tutorial or two, then jump straight in and analyse a dataset. They get a plot, maybe a few statistics and voila! They are now scientists who can code.

Kinda.

See, the thing is there is more (much more) to implementing programmatic workflows in a biological context. Unfortunately, these are the skills that intro-to-programming courses often fail to teach you (or sometimes even mention) - but without them, adding scripts to your workflow can often make your work less reproducible, not more. For the sake of the argument, lets pick one.

Version control.

Imagine a bowl of spaghetti (lots of lines of code), which you admire briefly before throwing at a canvas (your interpreter). You create a beautiful work of art (your plot), and stand back to admire your handiwork. Maybe you show it to a few friends, and they are super impressed. (Awesome!). And then they ask - how did you do it? Not just the throwing motion, or the rough distance between the bowl and the canvas. They want to know the precise location of every single strand of spaghetti in the bowl before you threw it at the canvas. They want to know the recipe you used for the sauce, down to the precise number of grams of oregano.

Now, of course, we aren’t actually talking about a bowl of spaghetti - we are talking about your code. So - of course - you can simply show them the script! Huzzah! The precise instructions that enabled you to make that very specific plot. But here’s the kicker. The code is not static. In fact, chances are that you poured over it for hours (maybe days) adjusting, tweaking, changing, testing and rerunning - all to get the glorious, awe-inspiring plot at the end.

By this stage, you’ve probably made over 100 spaghetti-splattered-spectacles (plots) and your house is starting to smell like an Italian Pizza joint. If I asked you to produce the exact recipe, down to the very position of each strand of spaghetti, for plot number 47 - could you do it?

My guess is probably no.

Conducting experiments is at the heart of science. However, without the essential extracurricular activities like maintaining backups, organising your results files and optimising project management systems, you wouldn’t last very long in academic research. The same is true of coding. Learning to write a functioning script is just the tip of the iceberg when it comes to implementing reproducible computing workflows in biology.

So, with this in mind, what are those all important skills you ask? Well, luckily, far more experienced scholars than I have written on this topic and provided guides (like and ) for those starting out in the computational space. These are highly applicable to biologists learning to code, and I encourage you to check them out before you stray too far down the spaghetti-on-the-wall path (like I did). As with any system, it is easiest to implement new routines before you have become to set in the old ones! At the very least, here are a few things to consider:

, including Github/GitLab/BitBucket
, including efficient folder layouts
and docker containers
practices
, including repositories and citing software you use

For more, make sure you check out the resources below.

Oh, and one last thing.

Unfortunately, this type of organisational work is not typically measured as a key performance outcome for biologists. Your number of git repositories or test coverage for an analysis suite is unlikely to come up in an award nomination or promotion application. And yet, it is entirely crucial as we move toward bigger and more complicated data and analyses - and so I encourage you to take the time to learn anyway. Maybe one day it will be recognised as essential work by the powers that be - but until then, at the very least, it remains essential for anyone wanting to do good reproducible science. And who doesn’t want that?

Final thoughts

So you want to learn to code? Yes - do it, 100%. I honestly couldn’t recommend or encourage it more. But please do so mindfully. If you want to have the best possible chance of integrating this wonderful tool into your scientific ecosystem in the long-term, you have to lay solid foundations and develop sustainable practice/methods.

Have you used any of these techniques as a budding bench-to-bytes biologist? Find me on or head over to the to tell me more!

Resources:

Image credits: Tomwsulcer via & olamishchenko via

Toolbox Talk: VSCode and Conda

Mon, 13 Apr 2020 07:51:36 +0000

There have been many rapid changes to how, and where, we work as the world settles into a pandemic-induced hibernation. After coming to terms with the fact that many labs could be closed for several months, life scientists around the world are increasingly resolving to make the most of their ‘downtime’. For some, this means upskilling in leading data analysis tools and techniques - and of course, I whole-heartedly encourage this initiative!

In my experience, this often takes two forms - those who use a course to learn and tackle generic coding projects, and the learn-by-doing camp who google as they go and tackle their own projects. I was a bit of both: using a few tutorials to understand the basic concepts and the remainder by going to work on real analysis problems. With that in mind, I thought it might be helpful to share a little about my setup and why a combination approach is useful (maybe even important) for self-taught data detectives.

One of the issues I have with generic online python courses is their often-overly simplistic approach to interpreted development environments (IDE’s) and package management. In my case, I had been dabbling in coding for a year or two before I came across the concept of virtual environments, and it took me another year or two to get comfortable in making, using and deploying these environments. Ditto with version control. Don’t get me wrong - I think it is great that scientists are diving headfirst into coding, and jumping into a course with a simple editor helps you hit the ground running. Unfortunately, this only defers the learning curve - if you are serious about integrating these tools into your workflow a grounding in the most basic of programming hygiene habits is essential. Shortcuts were necessary in days past, as installing and running python could be … cough … tricky for beginners. However, these processes have come along in leaps and bounds - meaning that it is now incredibly simple to get going with a basic python install in an editor that is easily expandable to incorporate, even encourage, basic code hygiene.

You may have come across my previous Toolbox talk about - these were the collection of tools I relied on to manage my coding, from writing to running to environments to version control. However, over the last 12 months, I have migrated to Visual Studio Code (VSCode) and Conda. I might be (a little!) biased, but this is a great place to start for anyone new to coding in python!

Conda: easy environment management

Before we dive into an editor, let’s start with creating and managing virtual environments. If you’re still not sure what virtual environments are and why you might need to use them, check out these great tutorials from and . In essence, virtual environments are essential for keeping your base install of python from becoming littered with packages, many of which might have dependencies that break each other.

Conda solves this problem for python and non-python dependencies alike, and it comes bundled with the ability to install various python versions. In fact, Anaconda and Miniconda have become the most popular Python distributions for data science in research laboratories worldwide. For a more in-depth discussion of these features, and whether you should choose the complete (Anaconda) or lite installation (Miniconda) check out the documentation .

Installating MiniConda

For many purposes, Miniconda is ample and is easy to and install by simply following the prompts. Real Python has an extra-detailed walk-through of the installation and starting your first environment in their fantastic , including creating your first environment and installing packages within that environment.

Once you’ve created your first environment using the conda create --name=my_env command, install the ipython and jupyter packages using the conda install ipython jupyter command. These packages will enable VSCode to locate and use this environment as a python kernel (more on that later!)

VSCode: a full-featured, user-friendly editor

Although technically VS Code is not a dedicated IDE, an abundance of out-of-the-box features supplemented with a few extensions gives you an editor that supports debugging, embedded Git, syntax highlighting, intelligent code completion, snippets, and code refactoring, among an abundance of other features. In short, it is incredibly customisable, user-friendly, language-agnostic, and seamlessly integrates with code hygiene tools to give every budding research software scientist the best start in their programming journey. In case I haven’t convinced you yet, here is a quick run-through from zero to coding hero with VSCode.

Basic installation

Download install VSCode for your operating system . Once installed, VSCode will open to the Welcome Screen. This is a great place to get familiar with VSCode functionality,including a number of helpful customisation portals (helloooo colour themes and keybindings!). For first-time users, your first stop should be the interactive playground. This window walks through some of the key features of VSCode and gets you started moving around the editor.

Getting started

The next stop is the Extensions tab, which is your gateway to much of VSCode’s fabulous functionality. You can explore the offerings and pick your favourites (there are extensions for almost anything you could want to do!), but the one must-have for our purposes is the Python extension.

python-extension

After installation, open the command palette using ctrl-shift-p and search for settings. In the settings search box, type “python” then scroll through the settings until you find the Python > Data Science: Send Selection to Interactive Window and make sure this setting is ticked. After this, you’re good to go!

python-setting

Piecing it all together: using your conda environment in VSCode

Now let’s get to work running your first python script in VSCode with your brand new environment! If you don’t have an existing python file to test out, not to worry we can make one!

Open a new file within VSCode, and save with a .py extension.
Inside the file, add the following line:
print(“Hello World”)
Save the file, then place the cursor in the ‘print’ line.

You should see in the bottom left corner of the editor that there is a python version displayed, followed by the name of your current environment. This will likely be the ‘base’ environment to start with. If you click on the environment name, then you should be able to select your new environment from the pop-up list. This will then be used to run the code selection.

To run your code within the built in interactive window, highlight the line and shift+enter

This will send the code to the interactive interpreter, which will open in a new tab within the VSCode window. The kernel may take a little while to start in the first instance, but you should see the Jupyter server details listed in the top right corner - note that it should be the name of your environment. Once it has run, it will look something like this:

hello-world

That’s it! You are all set to run python files for exploratory data analysis in VSCode.

Favourite features

Now we have mastered the basics, you have enough to get going with python in VSCode. A few features that make VSCode my one-stop-shop for all my coding adventures and beyond include:

The python extension itself! It is an evolving masterpiece - huge shout-out to the creators and maintainers - with a multitude of features packed within the interactive window that, frankly, could occupy a post on their own. Highlights include the variable explorer, ability to save the current state of the interactive window directly to a Jupyter notebook, and the ability to render a Jupyter notebook in a standalone tab where you can edit and run code cells without spawning a browser window.

The second important, must-use feature is version control. If you haven’t incorporated version control into your workflow yet, or aren’t familiar with it, you can read more . In short, even if you have never written a single line of code in your life, version control is a must for anyone routinely creating and editing documents of any kind. And the best news is, VSCode makes it only too easy! You can create repositories, then stage and commit changes from right there in the version control tab. More advanced functionality is also available in the extension.

Another handy feature is the ability to launch a command-line terminal within VSCode. This can be done via the “Terminal” menu, or keyboard shortcut ctrl+shift+` . As a bonus, if you have an active conda environment, this command will launch a conda terminal with the conda environment pre-activate on the command line. This makes it easy to install new packages and create new environments right from within the editor.

Last but certainly not least is the dynamic documentation display for functions derived from imported packages. For example, when using a the pandas melt function, typing pd.melt() prompts a popup which contains the function definition and variable explanation. This means no more googling docs to check that you have set all the required arguments!

VSCode also features all the best bits of Atom, like being able to set the colour theme and change the layout of windows, and a built-in debugger window that allows you to investigate the state of variable and monitor breakpoints in your code. The more you use and explore, the more functions you will undoubtedly find that make your life better. However, here are a few more of my favourites for inspiration:

Collaboration: (think googledocs for code)
Text functionality: , ,
Coding cheats: , ,
Prettifying the editor: , ,
Preview functionality: ,

Some final thoughts

I’ve gone through a number of IDE’s since my coding journey began. As a general rule, IDE’s suit my style of work better than command-line activities and the ability to run and interact with code and plots in short succession is essential for this. However, a good programmer needs at least some level of experience with the command line, so try not to avoid it! Luckily, VSCode makes transitioning between the command line and interactive windows so easy that there really is no excuse.

When picking an editor, getting comfortable with a single system setup can make it difficult to leap into something new - especially with the time and effort investment required to get everything customised and functional for how you work best. However, it is worthwhile keeping in mind that where the field moves as a whole is often where the greatest level of innovation will be (extensions, support etc). It is good to keep an eye on what the ’next big thing’ is, but don’t be afraid to wait it out until you are happy with the level of stability and usefulness to make the tradeoff worthwhile.

Last but certainly not least, don’t worry if there were parts of this post that sounded like mumbo-jumbo: everyone is at different stages in their coding journey. One idea I hope I have instilled is that VSCode caters to all skill levels, from the ease of installation and setup to the massive extensibility provided by the extension marketplace. In short, VSCode ticks all of my must-have boxes for an editor and I thoroughly recommend you try it out!

Do you have a favourite IDE for editing python code? Having trouble getting started in VSCode? Or maybe you tried VSCode and found it doesn’t quite suit? Get in touch via the or on - I’d love to hear about your experiences!

Image credits: via unsplash

BioInfoSummer 2019

Sun, 29 Dec 2019 09:28:44 +0000

With half my home state on fire and the lead up to Christmas, this post has been in the works for a few weeks. But here it is - better late than never! My brief recap of my recent adventure to BioInfoSummer 2019.

Conference details

Title:

Date: December 2nd - 6th 2019

Location: University of Sydney, Australia

Overview:

BioInfoSummer brings together advanced undergraduate and postgraduate students, researchers and professionals from the mathematics, statistics, medical sciences and information technology disciplines. Attendees develop bioinformatics skills, national networks and employability. The central themes of the 2019 conference were Epigenetics/genomics, Single cell omics, Mass spec analytics and BioCAsia/precision medicine.

Overall thoughts and impressions

Of all the conference travel I have done this year, BioInfoSummer was the closest to home and the furthest from a normal conference. The wide applicability of omics techniques meant a broad spectrum of research topics presented by a diverse collection of researchers with a common love for data analysis. In addition, being targeted at students and early-career researchers meant the dynamic of the seminar sessions was quite different - an atmosphere in which students were comfortable asking questions and driving the discussion around world-leading best practices in the field of bioinformatics. This was supported by plentiful and generous travel scholarships boosting the participation of interstate early-career researchers.

Even though my research has been edging into bioinformatics for a little while, this was my first formal foray into the community. While I was a little anxious about being immersed in a new group of people, luckily bioinformaticians are a friendly bunch! This meant that the workshops had a welcoming feel, creating a safe space to actively try out the techniques presented during the morning symposium sessions.

Last but not least, the location and timing of BioInfoSummer meant that they were able to share sessions and speakers with , and were followed directly by . This enriched the sessions and provided even greater diversity and networking opportunities. This truly was a fantastic blend representative of the bioinformatics scene in Australia. I cannot recommend enough for new PhD students or post-doctoral researchers new to bioinformatics in it’s many incarnations.

Lessions learnt

1. Bioinformatic techniques for proteomic analyses lags behind the trail blazed by genomics - this was evident in the distribution of the program, but also attendees. However, I also think that…

2. The distinction between proteomics and genomics is sometimes unhelpful - similar underlying data structures make some methods applicable to both, and leveraging the substantial ground that has been covered in genomics will assist in the rapid progression and development of proteomics methods.

3. Single cells are the future, but not the complete story - innovations in single-cell proteomics and ongoing developments in single-cell RNAseq will continue to push the boundaries of our understanding of biology on the smallest scales. However, one of my favourite talks of the conference was on the value of bulk methods in an era of single-cell capabilities. And I agree - these methods should be applied as appropriate and often one and inform the other.

4. Seeing is believing - two-dimensional imaging mass spec and spatial transcriptomics are the next frontier. With these techniques, we are starting to understand the spatial distribution and cooperation of cells in health and disease.

5. The future is now - the big data buzzword - everyone will have heard the term big data. This is becoming more the norm than the exception, even in biological research. This has meant that method development and bioinformaticians are in high demand, and become recognised as an integral part to any research project. This was exemplified by one of the conference keynotes, in which the speaker discussed the essential skills for a bioinformatician in the next five years; many of these included the ability to wrangle, access and store large amounts of data, as well as being able to leverage this data for biological insight.

6. Unexpected acquaintances - I travelled to Sydney expecting to know no-one. In a classic small-science-world moment, within five minutes of arriving I ran into a post-doc from another research group in my building! While we had met and talked a little previously, the chance to spend a week nerding out over data analysis and coding methods was the perfect way to get to know each other better. In fact, this was one of the best parts of the conference. As well as this, I met a handful of people from my alma mata who grew up in a similar corner of the world as I did.

Take home resources

There were too many great resources to share them all, but if you’re looking for tutorial style activities then the Material for each of the is a great place to start. Other than this, below are a few of the packages, tutorials or databases that I am keen to check out after hearing about them during the conference:

’s featured workspaces demonstrate common genomic analysis pipelines deployed using data and tools available on AnVIL.
is a portal to a series of public experiments describing mouse and human stem cells and how they differentiate to become mature cells, tissues and organs.
is a collection of lightweight and handy tools for Python that help to handle various sorts of proteomics data.
: A Python Library for High-Throughput (Phospho)proteomics Data Analysis
The repository contains a list of open source Python tools for Proteomics analysis. The list is very likely incomplete and we are happy to take pull request with new tools.
The workflow management system is a tool to create reproducible and scalable data analyses. Workflows are described via a human readable, Python based language.
workflow: differential discovery in high-throughput high-dimensional cytometry datasets

These will hopefully give you a flavour of the overall themes covered at the conference, but if you are new to the world of bioinformatics I thoroughly encourage you to get involved next year. The conference is slated to be held next December in Canberra, so keep an eye out for the official announcement!

Overall, BioInfoSummer was a great opportunity to connect and build community with like-minded people. I was so lucky to be supported by an AMSI ChooseMaths travel award - without their support, I could not have travelled to attend the conference and I cannot thank them enough for their support of Women in STEM. If you are new to or interested in the world of bioinformatics I whole-heartedly recommend checking it out next year!

Image credits:

Behind the scenes: Investigating the Investigators

Thu, 31 Oct 2019 01:30:27 +0000

The hottest topic on Australian researchers lips of late has been the recent NHMRC Investigator Grant announcement. I recently set about flexing my data-science muscles to see what the outcomes of the scheme were overall, and what a successful application might look like for the next round. If you haven’t come across it yet, you can read more .

Below is a behind-the-scenes look at what went into the data analysis for this post, and what I learnt along the way. I have also released all the code and results as a for those who like the real nitty-gritty details or want to extend the analysis.

Raw data

The raw data used in this analysis came from three main sources.

The first was the , which offers spreadsheet summaries for grants released since 2013. I chose to use only the data from 2015 onwards, for a couple of reasons: (1) the structure of the Fellowship system appears to have changed in 2014 to the ECF, CDF, RF layout which remained in place until 2018. This meant that 2013 data was poorly correlated with the more recent datasets. (2) The 2014 dataset did not have as much detail in the gender, age, state breakdowns that could be easily compared to the following years. (3) Five years seemed like a nice time period to work with!

The second source of data was the Field of Research codes used to classify research. You can find the complete list at the . I struggled to find an easily-downloadable version, and instead copied them from the University of Melbourne intranet. With a little post-processing, I had a fully functional list of each level of classification, which I could then use to understand which types of research were popular for funding.

The last source of data was , which I used to gather the number of research publications and average Field-Weighted-Citation-Impact (FWCI) for each awardee in the ten years previous to their year of award. This was somewhat of a manual process, and I used the ‘best match’ profile for each awardee imported into SciVal. Overall, 88% of the awardees were matched accurately (and this could be increased with a little manual curation). I also did a little digging around in the PubMed Central API using a python package (see the resources list below for more information) to batch-query the author names and collect their publication history, to compare with the matches generated by SciVal.

Processing and analysis

After initial cleaning of the raw data, I then equated the new and old schemes by matching the tiers. Although the correlation is imperfect (due to changes to eligibility between the old and new schemes), this resulted in Early Career Fellowships mapping to Emerging Leader 1, Career Development Fellowships mapping to Emerging Leader 2, and Research Fellowships mapping to Leadership Fellowships.

Lessons learnt

Data is messy. This was evident in all of the raw data I collected - naively, I expected simple-to-use spreadsheets from the NHMRC. At the very least, I was anticipating similar formats for the more recent 5 years. What I was greeted with was a complicated series of tables designed for visual interpretation by human eyes and not easy accessibility via computer scraping. The initial data cleanup took more than half the time it took to complete the analysis.

Through this process, it became clear to me that as a general rule people, despite often working in a science-oriented role, do not handle, label or store data well. With the increasingly data-driven world we live in, we would all benefit from improving our data hygiene.

Lastly, while the trends I saw and commented on are indeed interesting, they should be interpreted cautiously. The data that the NHMRC provides is somewhat fragmented (to protect the privacy of successful applicants). More importantly, the data they do provide is focused on successful applications. There are lots of important details about the makeup of the applicant pool that we don’t see and this is important - albeit missing - context for interpreting the trends I highlighted.

Similarly, publication history searches are a tricky one. Pay-walled publication information is a nightmare to access, and although PubMed searches are OK they are limited. To get standard metrics such as H-indices and field-weighted citation impacts requires access to subscription services. Moreover, people’s names are difficult! Even once you have access to the databases, it can be difficult to know whether/how to split the given names into first/last and there is almost no chance of avoiding manual curation if you want a complete dataset.

Tricks and tools of the trade

As this was my first dedicated data-science style project, I quickly ran across a few questions.

When wanting to plot the per-state distribution of applications and successful awardees, the most obvious visualisation was a map. I’d never plotted a map before, and after a quick google search I found myself asking What on earth is a choropleth and where do I find a shapefile? It turned out a chloropleth is what I wanted to make - a thematic map in which areas are shaded or patterned in proportion to the measurement variable being displayed on the map. And to do this, you need a shapefile - a vector data storage format for storing the location, shape, and attributes of geographic features. Luckily there are a few , and eventually, I found the I was looking for. Amazingly, using meant dealing with this type of data relied on many of the skills I already have and before I knew it Voila! - one map of Australia complete with colour-mapped and labelled data.

While handling the somewhat messy task of linking successful awardees to their publication track record, I came across the problem of slight variations between different naming formats. How do you match text that is fuzzy, and how does Levenshtein help? It turns out that Python has a whole range of functions via the that help to compare strings that are referring to the same thing but are written slightly differently. The most simple version of this makes use of the Levenshtein distance, named after Vladimir Levenshtein who originally considered this phenomenon in 1965. This metric measures how far apart two sequences of words are according to the minimum number of edits needed to change one into the other. These edits can be insertions, deletions or substitutions. One detailed later, and I was on my way matching fellowship awardees to scival authors in no time.

Finally, throughout the whole plotting process, I was conscious of my colour palette. I am a firm believer that good dataviz should be both functional and beautiful. With plenty of gender-based comparisons, you might wonder how I could go past the tried and true colour combination. So why I didn’t choose pink and blue for gender studies? It turns out there are a few reasons why red and blue for gender data is an ‘unawesome choice’ rooted in gender stereotypes (pink = girls = weaky, cute and blue = boys =strong, bold). Luckily there are plenty of good colour-combinations that circumvent these .

With these tools in hand, I had everything necessary to analyse and visualise the datasets at my disposal. To see these tricks and tools in action, don’t forget to check out the or head along to some of the resources listed below.

Resources

Wishlist: I haven’t had a chance to implement this functionality yet but these dataviz tools are high on my to-try list to extend the accessibility of this dataset!

Disclaimer

The original analysis was intended to inform my personal decision of whether to apply for an Investigator Grant in the 2019 round. Any action you take as a result of this information is done at your own peril. If you do decide to act on this information, I wish you the best of luck whichever path you may choose. May the odds be ever in your favour.

This being said, I have of course aimed to be as unbiased and informative as possible. This is also my first foray into data-science-for-public-consumption, so if you do notice any overt errors or bugs feel free to raise an issue on or get in touch via and I will check it out as soon as possible.

Image credits: cadop via

Investigating the Investigator Grants 2019

Mon, 30 Sep 2019 01:26:50 +0000

Introduction

The hottest topic on Australian researchers lips this month has been the recent NHMRC Investigator Grant announcement. After all the commotion around the new scheme and the distribution of research dollars, I set about flexing my data-science muscles to see what the outcomes of the scheme were overall, and what a successful application might look like for the next round.

What are Investigator Grants?

If you are a scientific researcher in Australia, chances are you were aware of the NHMRC announcement a few weeks ago of their latest round of funding. For those who are outside health-focused research, this was a highly-anticipated announcement after a complete re-structure of the NHMRC Funding Schemes over the previous year. The reform included replacing the Fellowship system (funds to support individual researchers to establish their own research programme) under the unified banner of Investigator Grants.

Historically, previous incarnations of these Fellowships (Early Career, Career Development and Research Fellowships) have supported the very best and brightest researchers in Australia with varying levels of lottery-luck in their award. Importantly, having been successful at the first rungs on the Fellowship ladder, researchers were then much more likely to gain the next level of support. Therefore, the award of these Fellowships to emerging researchers is a crucial support system that could make or break a career.

In particular, the lists better support for Early and Mid-Career Researchers (EMCRs) as a key goal of their reform:

Early and mid-career researchers are being discouraged from pursuing a research career.
\[…\]
In response to these issues, NHMRC reviewed the structure of its research funding and has reformed its grant program.

NHMRC, 2019

Why the fuss?

For researchers who spend many (many) weeks (months) working on the application for these schemes, the outcomes are of intense interest. The research funding environment in Australia is extremely competitive, and often the difference between funded and not funded can be little more than pot-luck. Early reports during the peer review process this year suggested that the new scheme had an abundance of issues and was likely to be even more of a lottery than previous years.

After the announcement, scientists flooded Twitter with their thoughts, interpretations and statistical analysis of the overall outcomes. There were many deserving people who were lucky enough to make the grade this time around. However, there were also an abundance of leading researchers who missed out and cursory analyses highlighted an abundance of inequities and unforeseen results of the reform.

_Sheldrick/status/1166908265743015936

_keating/status/1166886599482724352

Many of the numbers you will see online come from the key performance data released by the NHMRC (data on gender, age and state awardee rates has been made available since at least 2013). It is my firm belief that, as scientists, we should use science (including data science) to drive our decision making. The trends reveal biases/underlying issues in the grant scheme, and provide rationale/direction for additional reform. In addition, the profile of previously successful applicants can help researchers like myself understand how we fit within the scheme and, according to whether (or not) we fit the profile, inform our decision to apply in the next round (which is not that far away!).

Crunching the numbers

Most of the raw data used in this analysis came from the . I collected some extra data on the Field of Research codes and individual researchers from additional sources including the and .

After initial cleaning of the raw data, I decided to equate the new and old schemes by ‘matching’ the tiers as best as possible. According to a fact sheet provided by the , equivalent schemes are as follows: Early Career Fellowships map to Emerging Leadership level 1, Career Development Fellowships map to Emerging Leadership level 2, and Research Fellowships map to Leadership levels.

Due to differences in eligibility (according to years post-PhD) as shown above, the correlation is not perfect. But it serves the purpose of being able to compare as the NHMRC intended at least.

For the nitty-gritty details of the number crunching and visualisation techniques that went into the makings of this post, be sure to check out the post soon. If you’re simply here for the pretty pictures and insights, read on!

Key insights

Overall trends

At first glance, it appears as though the new investigator scheme has seen an injection of more money into the Fellowships portion of NHMRC funding. However, this funding includes more money per person at the upper tiers without a substantial increase in the number of applications funded. Moreover, the number of successfully funded applications and dollars was skewed toward the Leadership levels with an overall decline in the number of level 1 and 2 (Emerging Leader) applications funded.

The decline in the number of applications funded was matched by a substantial increase in the number of applications submitted, leading to a nosedive in the success rates particularly in the EL2 and L1 brackets. This increase in applications was thought to reflect the changes in eligibility structure meaning everyone had a go and often at levels far below what was appropriate for their research experience. The NHMRC is reported to be hopeful the number of applicants will decrease in the next round (thus artificially inflating the success rate).

Location, location, location!

With the conglomerate of research institutes, personnel and equipment in Melbourne, it has always been tough to beat in funding success. The same was true of this year, with Victoria the overwhelming leader in the number of awarded applications and second-highest success rate. In fact, Victoria received more than 45% of the awarded Fellowships in 2019.

The large number of applications originating in Victoria, coupled with their high success rate, suggests that not only is there a critical mass of outstanding researchers concentrated around the Melbourne biomedical hubs, but that they benefited immensely from the support processes in place in the lead up to submission. Hopefully, other institutes around the country can find value in their strategies for the following rounds.

Gauging the gender gap

Gender has always been a touchy topic when it comes to Fellowship funding in Australia, especially at the later stages of academia. There were concerted efforts made to target gender equality in the new scheme. However, at first glance, there was an enormous bias in the total number and dollars awarded. In fact, this disparity is the worst that the Fellowships scheme has seen in the last five years.

If we dig a little deeper, it becomes clear that this skew is due mainly to biases in the upper tier of the scheme. In fact, at the first and second tiers (level 1 and 2), there has been progress toward equality and, in some cases, even over-representation of women for the last five years. However, the successful males outnumber females two to one at the highest tier (level 3).

Two to one. The new scheme, if anything, has made this worse with the proportion of awards for women the lowest it has been since 2015. This points toward a systemic issue with how relative to opportunity is assessed and the lack of support for women entering the upper echelons of academia that is still ingrained in the research culture in Australia. Moreover, the extended 5-year term of these awards means that these differences will permeate University faculty for many years to come.

Titles and track records

As a general benchmark, academic titles say something about a researcher’s seniority. Considering the distribution of titles among successful awardees, there is a clear trend toward more and more senior researchers finding success at lower levels of the Fellowship scheme. For example, Associate Professors have taken a share of the level 1 funding for the last two years and for the first time, there was an Emeritus Professor awarded level 3 funding in 2019.

Similar trends can be seen in the track record of successful awardees, especially their publication history. While this is imperfect due to author name mismatches, overall the median number of publications for level 2 awardees has held steady around 50. In contrast, level 3 saw a sharp increase this year for the first time. The median number of publications for level 1 awardees has also steadily increased, meaning that to be competitive ECRs now need on average twice as many publications as they did 5 years ago.

With the revamp of funding levels and removal of specific ‘years-post-PhD’ ranges for level 1 and 2, this was always a concern. While I have (many, many) issues with using the years-post-PhD award as a ruler to measure relative success, at the bare minimum I do believe this distinction helped stratify junior researchers in the eyes of reviewers and assisted their assessment relative to opportunity. Unfortunately, the lack of boundaries this year left many researchers unsure of the appropriate level of funding they should apply for and moreover allowed many to take advantage of the lower levels in the scheme. This placed a large responsibility on reviewers to fairly evaluate an individuals trajectory against others with up to 10 years longer in research. A big ask!

Trendy topics

With every application, researchers include up to five keyword phrases describing the focus of their proposed research. By looking at the most popular keywords, I wanted to understand the research themes attracting the most funding and potentially consider how this has evolved over the last few years.

2015 keywords

2019 keywords

Interestingly, the top five keywords for the previous five years have typically been some iteration of health, disease, biology and cancer. While these seem very general, reading between the lines reveals a transition from ‘disease’-driven research to ‘health’-centric. Interestingly, epidemiology has emerged as a prominent focus this year. Cancer research has also held a steady proportion of funded applications, suggesting an area of high priority either among reviewers or impressive researchers embedded in this field.

What is unclear from these trends is the type of research being funded in these proposals. This is captured in the NHMRC reporting process as ‘broad research themes’. Traditionally, fundamental (Basic) research has held a large share of the total funding. However, the last five years have seen a steady decline in this proportion and a corresponding increase in the more translational themes.

In particular, clinical medicine and science enjoyed a sizeable bump in the proportion of funding awarded such that it has almost reached parody with basic science. This likely stems from the strong emphasis on ‘research impact’ that permeates all aspects of the new scheme, and at face value is not terribly alarming. However, our translational research must be underpinned by quality fundamental understanding and our ability to fund this type of research from the NHMRC scheme appears to be questionable in the future.

What does this mean for science, and ECRs, in Australia?

Overall, there has been a shift toward more senior and established researchers in all three levels of award. This is not merely a consequence of the new scheme, but has definitely been accentuated by it. This is likely reflective of the desire to invest in researchers who can demonstrate their previous impact has a direct and immediate public benefit.

This is also reflected in the share of the pie gobbled up by each of the four broad research areas, where Clinical and Medical Science has seen a steady increase at the expense of Basic Science.

While I made every effort to take a comprehensive snapshot of the available awardee data, these insights were limited in part by fragmentation of the data. The raw data provided by the NHMRC includes summary totals e.g. by gender or by state, but often the per-applicant information, due to obvious privacy reasons, is not available.

Some of the missing data, such as the years-post-PhD for successful applicants, could, in theory, be provided anonymously and would provide tremendous insight for potential applicants. At this stage, the closest proxy for years post-PhD is the mean age of awardees (although this as a measure is complicated by relative to opportunity). This year the mean age for level 1 awardees increased from 35 in 2018 to 37, agreeing with the overall sentiment of increasing seniority among successful applicants. With the restructuring in 2019, the NHMRC has all but done away with the ECR funding dedicated for those entering the postdoctoral workforce and require us instead to relying on the provision of project funding by senior researchers for many years before being deemed worthy and ‘impactful’.

There have been a number of statements released by associations commenting on this and other perceived failures of the new scheme, and suggesting changes to overcome these issues. For example, the released a host of recommendations aimed at alleviating many of the pain points ranging from the application structure to review processes. However, the short turnaround time (given applications for the next round open tomorrow and close in a little under two months) means that these recommendations are extremely unlikely to be implemented until next year. Moreover, this is a general and of research funding schemes world-wide. At this point, like so many other ECRs in my position, it is now time for me to think about whether my time, energy and effort are best placed in the hands of the NHMRC, other Australian funding schemes or - like so many in my position - whether I should instead focus on opportunities abroad. In the spirit of data-driven decision making, hopefully these insights have helped guide your thoughts as they have mine.

Resources

The original data was sourced from the
For more on the initial guidelines provided during the scheme restructure, check out the
Author track record information, including publication number and field-weighted citation impact, were collected from . If you are considering an application in the upcoming round, it’s a great idea to benchmark yourself against previous successful applicants.
For more info on the specific number crunching and data visualisation techniques used here, don’t forget to keep an eye out for my post.

Are you thinking of applying in the next round of Investigator Grants? Did any of these stats surprise you, or were they helpful in your decision of whether or not to apply in the next round? Head over to the , or let me know on .

Image credits: inspecting gears with magnifying glass | @ pluyer via

Toolbox Talk: Storing data using HDF5 files

Sun, 28 Apr 2019 05:43:45 +0000

As an early-career researcher, I am still figuring out the best way to get stuff done. Every day it seems as though my to-do list gets longer and longer, for every item I tick off two more take its place and I have no doubt it will only get worse. I am all about finding the best, most efficient and connected technologies to help alleviate some unnecessary legwork. I have decided to write more on the programs, apps and integrations that I use every day in this Toolbox series. This post focuses on how I store and organise the currency of science: big data!

Data: the invisible bits and bytes of modern life.

Data is an invisible and yet fundamental driver of our digital life in 2019, and the amount of data worldwide is growing every single day. In fact, the International Data Corporation (IDC) estimates the amount of data in the world will grow from an estimated 33 zettabytes (one zettabyte is equivalent to a trillion gigabytes) in 2018 to 175 zettabytes by 2025, an almost unfathomable amount of bits and bytes. Every industry is grappling with the growing mountain of data, and science is no different. Scientists use data as the fuel that powers insight, discovery, and innovation. This demands new infrastructure from the institutes in which we gather data; but it also demands new approaches to the way we as scientists collect, store and analyse data.

During the first five years of my university career, everything I needed for my degree fit safely on a single 16 GB USB stick. Then I went and learnt how to do single-molecule microscopy during the second year of my PhD, and suddenly I was generating 16 GB of data in a single experiment. To cope with this rapid explosion, I invested in the hardware needed to store this data as text or csv files and stumbled my way through establishing a folder-based filing system.

It was during this stint early in my PhD that I first got a taste of programming to deal with this data (for more, check out some of my other programming posts ), however, this still centred on csv/text files for importing data, calculating results, and exporting the final findings. This meant that the data along the way, including input, intermediate and final files, could still be opened by easily-human-readable programs such as Microsoft Excel, and kept supervisors/collaborators without programming experience comfortable.

Recently, this storage pattern was no longer adequate for the size of results I was dealing with. Enter: HDF5 format. If you’ve never heard of this format before, it’s somewhat similar to an Excel document, without the proprietary software tag. Read on for more about what it is, why it’s useful for those dealing with large datasets and how I implemented a storage workflow in HDF5 format using python.

What on earth is HDF5?

Heirarchical Data Format (HDF) is a collection of file formats designed to store large amounts of data in an organised manner. Similar to the way .txt refers to text files or .pdf refers to Portable Document Format files, HDF files are adorned with .hdf, .hdf5, .h5 style extensions. The ongoing development and accessibility of the HDF file format is maintained by the non-profit organisation “The HDF Group”, meaning the tools to store and use HDF files will never rely on proprietary (and often expensive!) software.

More specifically, HDF5 files consist of Datasets that can store arrays of data (think individual sheets in an Excel document), Groups which can store datasets or other groups (think a folder of Excel documents, or folders of folders), and metadata consisting of mapped key-value pairs for attributes of the data (think a detailed description notes page for each sheet/document).

Pros and Cons of the HDF5 format

As with anything, there are good bits and bad bytes about using HDF5 to store data.

Some of the Cons:

Unable to easily open HDF5 with excel or notepad (although there are some being developed that could help overcome this)
No inbuilt calculation or manipulation options
Not readily used by many researchers in the life sciences, including many senior researchers making it difficult to share data with collaborators unfamiliar with the format

A few of the Pros:

Heirarchical format allows for logically storing data in a single file with folder-like architechture
Allows pre-processed data, such as date-time data, to be stored efficiently without loosing the effect of preprocessing (as would otherwise happen in .csv format)
Fast. Compared to other file formats, reading and writing HDF5 files is speedy. For example, writing to HDF5 is 16 times faster than to simple csv file, not to mention the extra overhead that comes from Excel documents. For a comparison to other common formats, check out the
Storing metadata within the file architechture makes sure that these attributes are accessible to anyone wanting/needing to access this data at a later date, and doesn’t rely on the bundling of additional “description” files to make sense of all the parameters and conditions under which the data was collected.
There are advanced options to create datasets that can be edited (rows added etc), and those that are read-only. This is provides an extra level of flexibility that allows, for example, intermediate datasets to be appended to making sure the raw data is not inadvertantly changed.

Sounds great.. But how do I use HDF5 files?

If you have no experience with programming, to get started you can download , which is freeware available from the HDF Group that will allow you to open and edit HDF5 files. In the interest of full disclosure, I have never used this software but it seems to work similarly to graphical user interfaces like Excel (without the price tag!).

If you have a programming language of choice, chances are there are - my tool of choice is of course Python! Even within python, however, there are a few choices for how to interface with the HDF5 machinery. Given that I use Pandas most often for data wrangling, it makes sense to leverage Pandas’ built in support. This includes methods that allows dataframes to be written to directly to HDF5 files using a method provided by the dataframe itself:

df = pd.DataFrame({'A': [1, 2, 3], 'B': [4, 5, 6]}, index=['a', 'b', 'c'])
df.to_hdf('data_filename.h5', key='Key_1', mode='w')

This method also allows another object to be written to the same file, with a second key ’s':

s = pd.Series([1, 2, 3, 4])
s.to_hdf('data_filename.h5', key='Key_2')

In this case, the heirarchy is quite simple: two datasets are stored under the single file. To get your data back, it is simple to then read back into a dataframe:

new_df = pd.read_hdf('data_filename.h5', 'Key_1')

For storing multiple files in this manner (i.e. using key-dataframe pairs), I have created a utility function using the HDFStore module (another interface provided by Pandas) that collects dataframes that have been loaded into a dictionary, then saves them to a single HDF5 file:

def dict_to_h5(filename, dictionary, **kwargs):
 store = pd.HDFStore(filename)
 for key, df in dictionary.items():
 store.put(key, df)
 store.get_storer(key).attrs.metadata = kwargs
 store.close()

NB: using the HDFStore in Pandas requires PyTables v3.0.0 or higher. To update your existing Tables installation run the ``pip install –upgrade tables``` command in your terminal.

These are both simple examples in which there is a ‘flat’ data storage pattern. This is great for storing a few large dataframes, or sequentially processed intermediate dataframes for a single result. In most cases, however, you will find it useful to introduce an additional level of hierarchical organisation similar to the folder storage system we are familiar with from OS interfaces. To do this, the keys should have a file-path structure which lists the group hierarchy that the dataset is to be added to. For example:

hdf =HDFStore('storage.h5')
hdf.put('tables/t1',DataFrame(np.random.rand(20,5)))
hdf.put('tables/t2',DataFrame(np.random.rand(10,3)))
hdf.put('new_tables/t1',DataFrame(np.random.rand(15,2)))

This will yeild the following data structure within the single HDF5 file:

/new_tables/t1 frame (shape->[15,2])
/tables/t1 frame (shape->[20,5])
/tables/t2 frame (shape->[10,3])

Python also has other packages specifically for HDF5 manipulation, which have a few more specialised tools. If you’d like to know more about these packages, or how to generate more complicated heirarchies, check out the list of resources below.

A few final thoughts.

Data is a part of life as a scientist, and as life scientists we must strive for better, more versatile approaches to storing and handling the ever-growing datasets produced by our experiments. For me, HDF5 has provided a simple and elegant way to interface with my data. It allows me to store raw and computationally-expensive intermediate ‘checkpoints’, and also means that I can maintain a single HDF5 file that combines data and metadata in a single place. The barrier to entry is quite low, and there are plenty of versatile ways to access data stored in HDF format.

Resources

A to HDF5 with Pandas
You can find additional utitility functions that make it simple to create and store HDF5 files for my projects in my package on GitHub.
For more complicated heirarchies, check out
Some other Python HDF utilities include , for which there are some great on how to generate structured heirarchies.

Do you have a favourite way to store, access and manipulate large datasets? Get in touch on and let me know!

Banner image credit: @ gabons via unsplash

A year in review: 2018 edition

Tue, 15 Jan 2019 09:52:30 +0000

The last few weeks have seen a flood of people posting their 2018 achievements, promoting the academic currency of papers published, grants won, students mentored and conferences attended.

Without taking away from these achievements, I wanted to focus on viewing myself as more than my work, and especially more than the ‘publish or perish’-driven manuscript list. And so, here lives my reflection on the year that was. Lessons remain a crucial part; after all,

“If a man looks at the world when he is 50 the same way he looked at it when he was 20 and it hasn’t changed, then he has wasted 30 years of his life,”

Muhamed Ali

Of course, I would be remiss not to celebrate the high points - but I have tried to temper the ‘gloating’ with a few honest low points. Finally, I take a look to the future. By defining goals which will drive my personal and professional life in 2019, I hope to kick off a fulfilling and smooth-sailing year with the course charted and the wind in my sails.

Wins

A snapshot of the high points

I received my first personal scientific project funding through an Early-Career Researcher Grant
I had four papers published, including the last of my PhD research, and landed travel funds for an international conference
I was priviledged to give too many conference presentations to count (Lornes, ABS-ABA, Winter School..)
I developed and grew this page, with 20 blog posts including tools, conference reviews, book reviews and scicomm
I produced hundreds (maybe thousands?) of lines of brand new code - but more importantly, saw a significant improvement in my ability and confidence when programming which has allowed me to assist others
I attended my first PyLadies meetup, where I met people like me (in science, learning python independently) and some less like me (employed full time in programming or data analysis positions)
I developed closer relationships with family and friends, as I worked harder to maintain and build these ties
I read two more books than last year
I consistently, regularly exercised, losing 15 kilograms
I hit a saving milestone and we bought a house

Losses

A snapier shot of the low points

I worked for two months on experiments that were to resuscitate one of my major projects
I’m now left with no papers in the pipeline, leaving lots for the year ahead to wrap up these projects
My coding projects are in somewhat of a state of disarray, in desperate need of a tidy and some documentation
I only read 2 books, and put 8 kilograms back on
I spent more than 50 hours writing six grants, all of which were unsuccessful
I was quick to judge others and less approachable than I would like to be, letting my stress seep to the outside
We unbought a hous

Lessons

The bits I wish I knew at the start of 2018, but had to learn the hard way

Have the hard conversations - with your boss, your partner, yourself
Self-care is about what you need - not what everyone else’s definition of rest and relaxation tells them (and they, therefore, tell you) you should be doing
“There are less qualified people than you doing what you want to do, just because they decided to believe in themselves. Period.”
Make a positivity folder and actively collect the warm fuzzy moments: emails, photos, messages, compliments, congratulations…
If you want to do something, really want to, then make time. Do it. No one else is going to do it for you, and your schedule will not magically clear. Life is never going to get less hectic or stressful.
Every day, hour, minute you spend wishing your way to the next thing (job, house, friend, partner) you will equally spend wishing you had taken the time to value what you had, to revel in it.
Prioritise, prioritise, prioritise

Goals

Hopefully to help me meander with purpose through 2019

It might be a little meta, but a big goal of mine this year is to learn to set SMART goals. Not so much “new years resolutions” - but personal and professional goals that I would like to work towards every day.
If you’ve never heard of the SMART goal concept, you can find a more detailed description , but in a nutshell SMART goals are:
- S - specific, significant, stretching
- M - measurable, meaningful, motivational
- A - agreed upon, attainable, achievable, acceptable, action-oriented
- R - realistic, relevant, reasonable, rewarding, results-oriented
- T - time-based, time-bound, timely, tangible, trackable
With this in mind, I have the following goals for 2019:
- Yoga at least two days in a fortnite during 2019. A flexible body -> flexible mind -> clarity of ideas, increased alertness and engagement with the task at hand. I would like to be able to better calm my mind when distraction threatens. I will track this goal via a calendar which records each session.
- Land the next job - be strong enough to throw myself into the applications and let go of the guilt of not chasing (limited) Australian funds. This will be measured via my application for at least 3 more Fellowship programs.
- Attitude adjustment - I will track this goal via weekly mini-reflections, where I will focus on (1) Learning to take control of what is within my reach, and to more rapidly accept that which is outside my control and move on, (2) Doing one self-positive task per day - learning, physical activity, kindness, and (3) Fostering self belief to replace self doubt, in personal relationships, professional settings and long-term career prospects
- Prioritise, prioritise, prioritise!

And with that, it’s time to say sayonara 2018 - bring on 2019.

May science be kind to you!

Image credit: @brookelark via

Toolbox talk: Atom, Jupyter and Hydrogen

Sun, 23 Sep 2018 11:47:32 +0000

As an early-career researcher, I am still figuring out the best way to get stuff done. Every day it seems like my to-do list gets longer and longer, for every item I tick off two more take its place and I have no doubt it will only get worse. I am all about finding the best, most efficient and connected technologies to help alleviate some unnecessary legwork. I have decided to write more on the programs, apps and integrations that I use every day in this Toolbox series. This post focuses on what is currently the best (in my humble opinion) hackable text editor for writing, programming and data analysis: Atom!

The basics: Atom as a building block.

Billed as the “hackable text editor for the 21st Century”, is a free, open-source, cross-platform desktop application for editing plain text and source code. While the base program was developed by GitHub, many of best functionalities are community built and maintained under free liscences. This combination of core support and extensibility has resulted in an unparalleled level of customisability, and makes Atom my editor of choice for everything from simple text documents, to data analysis to developing python packages. For coders and non-coders alike, there is something for everyone in the Atom ecosystem.

From an aesthetic point of view, Atom has a nice panelled interface which can be extended at will. Below you can see a standard layout I gravitate towards while I am analysing data. The leftmost pane (the file tree) allows you to open an entire folder or folder tree to easily see all of the files associated with the project you are currently working on, complete with coloured file-type icons for easy visual grepping. In addition, I can have multiple panes open side-by-side which allows me to operate on code/data in one building and document the analysis approach in the second. Finally, to the right is the output/Git pane (more on that later).

An example analysis layout in Atom.

Atom also has a vast array of installable themes to customise the window colour and aesthetic. I myself am partial to a dark theme, with a subdued syntax highlighting (check out the OneDark theme). If you prefer a more traditional light theme, there are plenty of those too!

The best bits: Git(Hub), Integrated Development Environment and Real-time Collaboration functionality

Being built by GitHub, it is fair to expect some level of integration with Git. If you’re not yet familiar with Git, or version control in general, you can read more about it . In short, even if you have never written a single line of code in your life, version control is a must for anyone routinely creating and editing documents of any kind. For now, suffice to say that Git has traditionally been a command-line tool and therefore mildly intimidating to those of us who found it with little experience. Recently, GitHub introduced a which alleviates some of the hassles of interacting with Git via the command line. However, Atom takes this functionality and seamlessly integrates it within the environment where the work is being done. No longer do I need to switch windows to another app or terminal - I can immediately see the documents that have detected changes, stage and commit those changes and push to GitHub from within Atom. More than this, Atom allows you to create and change branches, and push/fetch/pull from remote repositories all without ever having to change windows.

One of the reasons I came to it in the first place was the flexibililty to create an Integrated Development Environment (IDE) for any programming language. An IDE typically provides comprehensive facilities to edit, build and debug source code for a given language. Atom has taken on this task to provide support for a long long list of the most popular languages. This is achieved via installation of community-driven packages - more on this in a little bit.

Finally, Atom provides an interface for real-time collaborative coding. While I am yet to test this functionality, Atom’s “Teletype” promises the ability for users to share their workspace with team members and collaborate on code in real time using a shared portal. If you have used this feature, be sure to let me know how you find it!

Easy extensions: Size may not matter, but it’s all about the package(s)

They don’t call Atom the most hackable text editor for nothing! Although the Git interface is packaged as part of the raw Atom install, many of the other features I’ve already described rely on the installation of community developed and maintained packages. The variety and functionality of these packages is seemingly endless, and installing a new package is as simple as opening the settings view and searching for the package. No matter what language you intend to use Atom for (or even just for general editing of plain text or markdown documents), there are bound to be a few packages that will make life easier.

Given Atom is agnotistic to language, there are plenty of packages for common languages like R, Java, C #…. If any of these is your language of choice, I encourage you to poke around online and see what packages people recommend. For the remainder of this post, I will focus on how I set up Atom as a home for my Python development and data analysis.

Accessorising Atom: a Python perspective

When first learning python, I used Spyder as my development environment. After a while, I got itchy feet and investigated Sublime (another text editor) and was intrigued by it’s customisability. After a brief stint, it wasn’t long before I discovered Atom and, as they say, the rest is history! There are plenty of generic “how-to-set-up-Atom-for-Python” articles (you can find examples and ), so I won’t spend too much time rehashing these here. Suffice to say, there are a few basic packages you will want to install to get the most out of Atom for your python projects. A few examples include:

Perhaps my favourite aspect of Atom (and in fact the reason I know recommend it to everyone I help get started with python) is the easy integration with Jupyter. If you haven’t come across Jupyter yet, Jupyter Notebooks provide an open-source web application that allows you to create and share documents that contain live code, equations, visualizations and narrative text in a whole host of languages. Notebooks support over 40 programming languages at the time of writing, including Python, R, Julia….This is managed through the installation of different , and if you are keen to run additional languages then check out how to access different kernels . In the case of python, IPython is the central dependency of Jupyter and thus is preinstalled. Jupyter Notebooks are in and of themselves a great environment for data cleaning and transformation, numerical simulation, statistical modelling, data visualization and much more. However, they require the analysis, version control and file management to be completed in different locations (other windows, apps, etc). Enter . This neat package brings all the functionality of Jupyter notebooks (documentation, in-line plotting and results etc) to your atom environment using the installed kernels. In my opinion, this is a must have for someone getting started with Python in Atom. To get started, follow the installation guide . As a bonus, is an extension package providing variable exploration in beta. If this is an IDE feature you cannot live without, check it out!

Better than basics: integrating python virtual environments

Similar to Git, using virtual environments with Python (despite being important best practice!) can be extremely intimidating for a newbie. Again, having to make and activate virtual environments via the terminal, and keep track of potential leftmost to use, set the bar above what I could digest and integrate into my workflow initially. Without this essential tool, my base install of python became littered with packages, many of which had dependencies that broke other packages. And don’t even get me started on the nightmare that was needing a safe install of both Python 2 and Python 3 for different projects!

Eventually, I got my act together and installed both and . If you need a little more advice, check out these great tutorials from and . For a little while, I was content that I could activate different environments if I desperately needed to - but was not overly inclined to use them on a regular basis! That was until I found the Atom package specifically designed to integrate this functionality: . This package lets you activate different virtual environments from within Atom, and combined with Hydrogen’s kernel manageraesthetic can have complete control of the python environment within which you are trying to run code!

Conclusions

Atom is a great multipurpose ecosystem with huge extensibility to tackle common tasks, both for general text documents and coding projects. There is support in the form of community developed packages, and complete customisability. By combining a few of these packages, it is possible to build a friendly development/analysis environment for those new to coding, or seasoned veterans alike, which encourages best practices at every turn.

That’s all from me for now - happy coding!

Quick Recap: PyLadies Melbourne Event

Tue, 31 Jul 2018 11:24:05 +0000

Locating the labyrinth a nd meeting the meetup For a few months, I had been working on the courage to check out one of the many coding-related groups held all across Melbourne each month. There are the meetups, various networking events and more data science groups than I could possibly list here (if any of these peaks your interest, check out the meetup list ). Still, as a scientist-turned-programmer (and a shy one at that) I wasn’t too quick to jump in the deep end with a roomful of people who of course I assume know everything there is to know about programming. Eventually, an email popped into my inbox with details for an upcoming centered around a tool I had been using (and hating) almost daily, and finally, I thought, “This is it. It’s now or never”. So, a cold Melbourne Monday night I bundled up my backpack and headed into the city after a long day at work.

After a tricky start (getting into locked business buildings in the heart of the Melbourne CBD is not my strong suite - mastermind criminal I am not!), I finally made it to a table jam-packed with Souvas accompanied by a fridge full of beer and cider (or apple juice poppers if that’s what you’re into). I was one of the last to arrive, so I quietly slunk in the back and collected some food before introducing myself to the orchestrator of the event. Not long after a quick round of “What’s your name, what do you do…”, we were bundled through to a second room where there was a bunch of tables and chairs set up around a projector.

For the next 90 minutes, enthralled and entertained with her tales learning python’s premier plotting library, matplotlib. I won’t go too much into the finer details of her talk - you can check it out . But, I will tell you a little about Ash and why her talk being my first foray into python meetups was so important. A masters student come research assistant, Ash is a bench-trained scientist with a flair for uncovering transcriptional signatures of immune genes in cancer. She came into python as the only scientist in her lab using this language and has built analysis pipelines for her work from the ground up. Coming into python as a scientist was definitely something I could relate to! Ash was brutally honest about how hard she found it at times to navigate the various Matplotlib interfaces. She unashamedly admitted how easy it was to spend hours ‘prettifying’ your graphs. And most importantly for me, she had a room full of fellow pythonistas, mostly software engineers and data scientists, nodding along. It was unbelievably reassuring to hear that I wasn’t alone in my experiences of matplotlib-induced pain. On top of that, having someone methodically explain the basic concepts, the tricky syntax and the bizarre labeling conventions was so helpful for crystallising the knowledge I had slowly accumulated over the last year or two since starting to use matplotlib more regularly. I even learned a few new tricks (aspect=‘auto’ anyone??).

I know it may seem like I’m gushing (and maybe I am, a little) but it truly was a revelation to me that scientists could have something so useful and relevant to contribute to what (on the surface) seemed like a simple conversation for those ‘in the know’. The PyLadies are an incredibly welcoming bunch, and I can’t wait for next month to ‘meetup’ with them again.

The nitty gritty When: Third Monday of each month Where: Melbourne CBD Who: Anyone identifying as female who is interested in python. What: The presenters and topics vary by month, so for all the details make sure to check out the PyLadies meetup page.

Note: The details I’ve included here are for the . If this is not near you, don’t dismay - there are PyLadies groups all over the world and most likely one near you! There are even remote PyLadies groups that regularly post talks online if this sounds more like your jam head to check it out.

Python not your thing? There are so, so many different groups on , I am sure you will find something that takes your fancy!

Resources If you’re in a world of matplotlib pain, here are a few of my favourite resources that (along with Ash’s talk) really help to lay out ground rules and will hopefully give you a fighting chance:

a weekend walk around the Bokeh library

Sun, 13 May 2018 11:13:53 +0000

If you’ve read my , you will know that I (somewhat arrogantly) consider myself “a passionate advocate for the use of programming by life scientists”. This came about following my Cambridge trip during my doctoral degree, in which I began to code analysis programs to derive meaning from the large (tens of gigabyte) datasets that we were collecting on a daily basis. At the time, I first learnt IgorPro - a proprietary language used by the other members of my host lab. While it was a steep learning curve, I was hooked. The power of being able to analyse huge reams of data without a single copy-paste-formula manouver normally completed in spreadsheets? Sign me up! Take out the human error of transferring data from program to program (Excel, GraphPad…) to achieve a result? Tell me more!

After my initial introduction, I quickly went in search of a more versatile language to learn. I disliked the feeling of being locked into the facilities provided within the commercial platform, and was tired of not being able to easily find support for this program online. And so, I met Python. And boy, did we hit it off! Python is easily the most friendly programming language I have come across, it is versatile and open source, the community is amazingly friendly, and I am yet to come across a problem I can’t solve via a quick trip to StackOverflow (seriously, where would I be without you guys?).

Little by little, I have worked on building my skills and I learn a little more each day. For a while now I have been comfortable writing small scripts that assist in my data analysis and plotting needs. However, I was yet to venture into the world of interactive plotting. I have always been intrigued by all of the different ways we can visualise data, and believe interactive plotting is a great way to allow both myself and someone else to explore a dataset. With this in mind, I set out this weekend to explore an interactive plotting library in python - Bokeh.

“Bokeh is an interactive visualization library that targets modern web browsers for presentation… Bokeh can help anyone who would like to quickly and easily create interactive plots, dashboards, and data applications.”

Sounds pretty good right? So, I gave it a go. As a test dataset, I decided to recreate the scatter plot from a published by our group for which the supplementary data is readily available. This plot displays the proteins whose cysteine residues significantly changed their reactivity following treatment with the drug Tunicamycin (an inducer of endoplasmic reticulum stress). Below you can find my version of Figure 6b, constructed as it is displayed in the article using Excel and GraphPad.

To recreate this plot in python, I took the raw data and constructed the scatter plot, colouring each datapoint according to the level of significance (as determined by the p-value). I also mapped the gene names associated with each datapoint via the UniProt API (a second learning exercise for this weekend!). This provides a second layer of data which I used to create the interactive feature of the graph - instead of displaying static names for only a few of the genes (which is limited by the need to not overcrowd the plotting area), gene names and ontology information is displayed by hovering over the point of interest. My final product is below - go ahead, give it a try! I found it super simple to embed the output here, plus there is also some handy toolbars provided by Bokeh out of the box that allow the plot to be saved to png or svg from the current view.

Bokeh Plot

I must say, I have been thoroughly impressed so far. Bokeh makes it extremely easy to build simple plots from existing pandas dataframes (think: spreadsheets) and to overlay interactivity without too much extra hassle. It offers the ability to completely customise the plots and layouts with widgets and extra functionality, although I think even the standard plotting elements I used here look pretty stunning in comparison to the original version!

So, here it is - my first passionate post about the benefits of programming for life scientists. Even if you take away the amazing benefits of reproducibilty, shareability, scalability and efficiency - the sheer ability to easily produce stunning visualisations that can be shared is a great asset. If you haven’t jumped on the programming bandwagon yet, what are you waiting for??

Advanced Scientific Programming in Python Summer School

Thu, 25 Jan 2018 23:10:08 +0000

Another post, another conference!

This time it was the Asia-Pacific “ . A full 6 days of intensive training in the programming language Python, designed to extend my amateur skills and give me more tools to analyse my data.

And boy, did it deliver - kicking off with some free swag!

The formal sessions started with a welcome from the school’s founder, Tiziano, who’s first iteration of the school was over a decade ago. His opening remarks really resonated with me:

In industry there are normally teams tackling a problem. and it is important to be kind to your teammates by writing nice code and documenting well. In science, we often work alone on our own data, and that teammate is your future self. This week is all about being kind to your future self.

Each of the main sessions targeted a specific element of scientific program in python (from GitHub to tidy data in pandas to plotting with MatPlotLib to parallelization to packaging to cython). The tutors would spend a few minutes making sure our basic knowledge was up to speed, before taking those concepts and extending them beyond what I had ever considered possible from my programming life!

A huge part of the success of the school was pair programming. The school provided one computer between two, all formatted identically and containing all the necessary applications for the week, with which we were to work in pairs to solve the lecturer’s exercises. This was a completely new experience for me - not only had I never worked with anyone else collaboratively in a coding environment, I was not accustomed to sharing my code with anyone! I have always been shy of sharing my programs, feeling like they were amateur and ‘half done’. Now, not only did my partner get to see the ‘final’ product, they would get to watch my thoughts take shape on the page! As terrifying as it sounds, I really did learn a lot from observing others and from their input as we worked together.

To consolidate our knowledge, on the final day we were split into teams where we would collaborate using our newly minted GitHub skills. The goal was to program two pacman bots which would be pitted against the other teams in a tournament - not only did I have a great team, and a great time learning to work with the bots, but we WON! This meant the delivery of swag #2: a signed copy of “Elegant Scipy”!

With tutors from Europe, New Zealand, South America and Australia, we were spoilt for expertise. Most surprisingly (and delightfully) of all was that the student profiles were just as diverse! Over the course of the conference, I worked with a particle physicist, a plant biologist, an engineer, a medical radiation physicist and a virologist. Not to mention the age, cultural and gender diversity - the group of students was a beautiful reflection of the diversity of the scientific community.

I can honestly say I felt so privileged to be selected (the conference was free to a limited number of students), and it was hands down the most practical and valuable conference for developing skills I have ever attended. If you ever have the chance to go, I cannot recommend it enough.

Happy programming!

Coding | DEZERAECOX

Learning to code: it's as easy and as complex as that.

Learning to code: it’s as easy and as complex as that.

Final thoughts

Resources:

Toolbox Talk: VSCode and Conda

Conda: easy environment management

Installating MiniConda

VSCode: a full-featured, user-friendly editor

Basic installation

Getting started

Piecing it all together: using your conda environment in VSCode

Favourite features

Some final thoughts

BioInfoSummer 2019

Conference details

Overall thoughts and impressions

Lessions learnt

Take home resources

Behind the scenes: Investigating the Investigators

Raw data

Processing and analysis

Lessons learnt

Tricks and tools of the trade

Resources

Disclaimer

Investigating the Investigator Grants 2019

Introduction

What are Investigator Grants?

Why the fuss?

Crunching the numbers

Key insights

Overall trends

Location, location, location!

Gauging the gender gap

Titles and track records

Trendy topics

What does this mean for science, and ECRs, in Australia?

Resources

Toolbox Talk: Storing data using HDF5 files

Data: the invisible bits and bytes of modern life.

What on earth is HDF5?

Pros and Cons of the HDF5 format

Sounds great.. But how do I use HDF5 files?

A few final thoughts.

Resources

A year in review: 2018 edition

Wins

A snapshot of the high points

Losses

A snapier shot of the low points

Lessons

The bits I wish I knew at the start of 2018, but had to learn the hard way

Goals

Hopefully to help me meander with purpose through 2019

Toolbox talk: Atom, Jupyter and Hydrogen

The basics: Atom as a building block.

The best bits: Git(Hub), Integrated Development Environment and Real-time Collaboration functionality

Easy extensions: Size may not matter, but it’s all about the package(s)

Accessorising Atom: a Python perspective

Better than basics: integrating python virtual environments

Further Reading: Example Atom/Jupyter/Hydrogen/VirtualEnv Setup

Step 1: Checking your default Python install

Step 2: Install virtualenv, virtualenvwrapper

Step 3: Configure virtualenvwrapper

Step 4: Create virtual environment

Step 5: List, activate and deactive virtual environments

Step 6: Install relevant packages for your new environment

Step 7: Install new ipython, and configure a kernel from within the environment

Step 8: Install Atom packages

Step 9: Using the virtualenvwrapper package in Atom

Conclusions

Quick Recap: PyLadies Melbourne Event

a weekend walk around the Bokeh library

Advanced Scientific Programming in Python Summer School