Scaling R Shiny: How We Built a Shiny App for 700 Users

<em>Updated on 7 September, 2021</em> Data science teams know only too well that building a working prototype is one thing, but scaling R Shiny and making your app production-ready is a very different story. Appsilon was the first company to demonstrate that a Shiny app can be used not only by dozens but hundreds of users. Such a pioneering project was presented by one of our senior data scientists and <a href="https://appsilon.com/data-for-good/">Data for Good</a> Lead - Olga Mierzwa-Sulima - at the useR! conference in Brussels.  Curious to learn how we managed to build a Shiny app for 700 users? Watch the recording from useR! or read on for key takeaways from the presentation. <ul><li><a href="#anchor-1" target="_blank" rel="noopener noreferrer">Scalability case study presentation at useR!</a></li><li><a href="#anchor-2" target="_blank" rel="noopener noreferrer">Challenges in scaling R Shiny</a></li><li><a href="#anchor-3" target="_blank" rel="noopener noreferrer">Scaling R Shiny apps in practice</a></li><li><a href="#anchor-4" target="_blank" rel="noopener noreferrer">Open source packages to help you build Shiny apps at scale</a></li></ul> <h2 id="anchor-1">Scalability case study presentation at useR!</h2> Scalability is a hot topic in the R community. Not surprisingly, Olga spoke at userR! to a <a href="https://twitter.com/matlabulous/status/882530484374392834">packed house</a>. All seats were taken and there were audience members literally spilling out the doors. <iframe src="https://www.youtube.com/embed/HcjqCn0hIGs" width="560" height="315" frameborder="0" allowfullscreen="allowfullscreen"></iframe> Olga’s talk was entitled<a href="https://channel9.msdn.com/Events/useR-international-R-User-conferences/useR-International-R-User-2017-Conference/How-we-built-a-Shiny-App-for-700-users"> ‘How We Built a Shiny App for 700 Users?’</a> She went over the main challenges in scaling a Shiny application and the methods we used to tackle them. The talk was partly in the form of a case study based on Appsilon’s experience of <b>building an app used daily by 700 users</b>. This, to our knowledge, was one of the biggest production deployments of a Shiny app at the time. For the latest achievements on the scaling front, tune into our <a href="https://youtu.be/MYVojGHeKAc?t=6194">R Shiny Masterclass</a>. <h2 id="anchor-2">Challenges in scaling R Shiny</h2> R Shiny is renowned for rapid prototyping of interactive web apps that support decision-making processes. It’s also a great tool for communicating and sharing research results prepared by data science teams. However, developing a Shiny app for a large scope project poses a challenge. Whenever a Shiny app is to be used commercially by more than a 100 users, R Shiny developers are facing major UI and UX demands. <h3>User expectations flying high for UI</h3> The first challenge is the user interface (UI). Being exposed to hundreds of web pages and applications, modern users expect each digital product to have a number of qualities. The user interface of a Shiny dashboard should have a slick design and a clear structure that’s easy to navigate. An intuitive access to all main features is a must. Containers grouping relevant content and color-coded labels help users quickly find the information they need. Even such details as the styling of tables can influence the ease of use and impact user adoption. <h3>User experience on the line</h3> <a href="https://appsilon.com/ux-design-of-shiny-apps-7-steps-to-design-dashboards-people-love/">User experience (UX)</a> is another big issue. Even the most spectacular user interface won’t be enough if the app is slow. At this point, the application performance directly impacts the user experience. Yet, scaling a Shiny app means accommodating a much longer list of requirements and a bigger user base. Maintaining efficiency and speed is crucial when selecting optimization methods. The key is to create a good UX with appropriate methods for scaling. There’s no need to sacrifice one over the other. <h2 id="anchor-3">Scaling R Shiny apps in practice</h2> The data science team building the Shiny app for the Boston Consulting Group started with just two powerpoint slides, but things moved fast from there. It took Appsilon only two days to build a prototype and two more weeks to have a working demo ready. Achieving such a pace of development wouldn’t be possible without an innovative approach. <h3>shiny.semantic for Shiny UI</h3> Appsilon’s <a href="https://github.com/Appsilon/shiny.semantic">shiny.semantic package</a> (an alternative to standard Bootstrap) is what gave Olga’s team the ability to build a beautiful and flexible Shiny UI. In essence, it’s a domain specific language used to wrap HTML tags. The package is open source, available on CRAN and <a href="https://github.com/Appsilon">Appsilon’s github</a>. With shiny.semantic, you can use semantic UI components and choose from different <a href="https://semantic-ui-forest.com/themes/">semantic UI themes</a> to create a graphic layer of any Shiny app. In a nutshell, shiny.semantic: <ul><li style="font-weight: 400;" aria-level="1">Downloads and imports semantic UI <a href="https://appsilon.com/howto-css-and-shiny/">CSS classes</a>.</li><li style="font-weight: 400;" aria-level="1">Creates an abstraction enabling users to define Shiny inputs (also text inputs).</li><li style="font-weight: 400;" aria-level="1">Contains predefined, complex elements as R functions that are ready to use.</li></ul> <h3>Performance boost</h3> Once the UI revamp was over, Olga and her team had to find a way to <a href="https://appsilon.com/speeding-up-r-shiny/">improve the app performance</a>. Appsilon analyzed the backend of the app to identify the processes that were affecting the user experience. Some non-standard optimization tricks implemented to gain performance included: <ul><li>Indexing using <i>data.table</i>: table search was 25 times faster than <i>dplyr::filter</i>.</li><li>Filtering in the frontend elements: a custom search algorithm gave users more flexibility.</li><li>Building a custom API with the use of <i>shiny::registerDataObj</i>.</li><li>Rendering hidden output: <i>suspendWhenHidden</i> set to TRUE*.</li></ul> *Please note that the last technique works in specific cases only. With too many hidden elements, changing this default setting might actually slow down the app. <h3>Complex reactivity</h3> Complex reactivity was another challenge, also related to component rendering. This issue is specific to Shiny and stems from the fact that the <i>reactiveValue</i> breaks the reactivity chain if a value doesn’t update. To mitigate that, we used the <b>reactive trigger</b> to force rendering of selected frontend components, but only when dealing with a change in output. This way, no unnecessary rendering caused by user input would affect the app performance, effectively <a href="https://appsilon.com/r-shiny-faster-updateinput-css-javascript/">making the Shiny app faster</a>. What’s interesting, it was Joe Cheng, the creator of Shiny, who introduced this simple concept. It enables you to connect two elements of the Shiny app, so that any change introduced in the one element triggers a change in the other element. <h3>Implementation and deployment</h3> Making sure the app is always available to the growing numbers of users was the final stage in scaling the Shiny app. Fortunately, <a href="https://appsilon.com/alternatives-to-scaling-shiny/">good app architecture</a> can do wonders in this respect. The first step to optimize implementation and deployment was to use a <b>load balancer</b>. With the load balancer in place, Olga’s team achieved a better distribution of workload across multiple servers. This meant housing multiple instances of the Shiny app in containers. It was important to make sure that each Shiny server had the same configuration and the same Shiny app run in each container. Additionally, we moved the authorization to a separate layer of the stack. The solution we used is compatible with Amazon EC2 and Microsoft Azure, but it can be also applied on-premise. The ways you can build a Shiny app at scale don’t end there. If you’re curious to learn how to leverage frontend, extract computations and use a database, watch this <a href="https://www.youtube.com/watch?v=iI5L8qmmZaU">video with tips for scaling R Shiny infrastructure</a>. <h2 id="anchor-4">Open source packages to help you build Shiny apps at scale</h2> The Boston Consulting Group was impressed with the results delivered by Olga Mierzwa-Sulima and the Appsilon team working on their Shiny app. Is your data science team dealing with similar challenges? Don’t forget there are a number of <a href="https://shiny.tools/">open source packages for R Shiny</a> that can help scale your Shiny app and give it the UI/UX it deserves. If you aim for superior Shiny dashboards, make sure to check out<a href="https://appsilon.github.io/shiny.semantic/"> shiny.semantic</a>, <a href="https://github.com/Appsilon/shiny.fluent">shiny.fluent</a>,<a href="https://appsilon.github.io/shiny.collections/"> shiny.collections</a>, and <a href="https://appsilon.github.io/shiny.router/">shiny.router</a>. Follow us on<a href="https://twitter.com/AppsilonDS"> twitter</a> to get updates about<a href="https://github.com/Appsilon/shiny.i18n"> shiny.i18n</a>: our open source package for internationalization.