Ending Software Boom and Bust: Lessons learnt from Scratchpads and proposals for building sustainable virtual research environments for the biodiversity research community

Abstract

Scratchpads launched in 2007 and became an extremely popular resource adopted for a variety of communities. Primarily, Scratchpads are used to manage and publish biodiversity data, but many sites were organised around projects, societies and regions. Demand for Scratchpads peeked at requests for more than 80 new sites over a 3 month period in 2014. Today we have over 1000 Scratchpad sites. This has not been a pain-free journey. In 2015 the grants funding Scratchpad support and development came to an end, and whilst the Natural History Museum, London, provided some institutional support, this was alongside several competing initiatives. For a period of nearly two years, the Scratchpads had no dedicated developers. The Scratchpads suffered from this neglect, bugs remained unfixed and the platform became increasingly unstable. This situation has now been rectified, in 2017 the Informatics Group expanded, enabling us to provide a dedicated resource to the Scratchpads again. This has been a challenging but valuable learning experience - and one that many Virtual Research Environments (VREs) in our community have, or will, experience. Current funding models encourage a boom and bust development cycle, described by Hine (2008) as the "dance of the initiatives," as projects constantly need to re-invent themselves in order to receive new external funding. We need to move beyond this, to start working collectively to develop a common roadmap for these systems, so we can begin to mutually benefit from each others development activities. Building on lessons learnt from the Scratchpads, we highlight a draft set of principles that may provide a framework for such a collaboration. While it is unrealistic to expect existing projects at different stages of maturity, and supporting very different use cases, to re-write their codebase in order to facilitate collaboration, we propose a microservices framework that would allow these related systems to converge on the delivery of a common set of services, provided by many current VRE's. This convergence, coupled with the development of a common and mutually agreed roadmap for these systems, has the potential to build a more sustainable future for VRE user and developer communities, as these systems evolve to support new use cases and improve existing functionality.