sparklyr

Data science in Spark with sparklyr

Quick reference guide for data science in spark with sparklyr

Navigating technology for data teams | Stanislav Seltser | Data Science Hangout

To join future data science hangouts, add it to your calendar here: https://pos.it/dsh - All are welcome! We’d love to see you! We were recently joined by Stanislav Seltser, VP of Enterprise Architecture and AI Infrastructure at State Street Global Advisors, to chat about how to strategically approach the business leadership about investing in new tools, forecasting technology obsolescence for data work, and fostering innovation and creativity in data science teams. In this Hangout, we explore the factors that can help forecast when a technology will become obsolete, including the rate of commits to open-source projects and the frequency of vendor releases. Stanislav also shares his approach to evaluating technology based on usability, cost, and maintainability, as well as how to ensure that chosen technologies provide a return on investment. Stanislav recommends treating the need for a new tool the way you would if you were pitching a start-up, with a business plan and a technical prototype, to really show your understanding of what you’re asking for. Stan also shared some solid career advice, so stick around for that. No surprise here, his advice starts with community - being social and connecting with as many people as you can - but he really emphasizes mentoring and helping others succeed. We agree, Stan! Resources mentioned in the video and zoom chat: Libby’s blog post on career advice from the Data Science Hangout → https://posit.co/blog/lessons-from-four-senior-leaders-in-the-data-space/ Databricks + Spark documentation for sparklyr → https://spark.posit.co/deployment/databricks-connect.html Rachael’s post on communicating the ROI of data science projects → https://posit.co/blog/tips-for-communicating-roi-of-data-science-projects/ Jim Hester’s talk on dependencies → https://youtu.be/mum13N7CGUI?feature=shared If you didn’t join live, one great discussion you missed from the zoom chat was a conversation around the struggles of getting out of Jupyter notebooks for developing coding projects in python and R. Do you love notebooks? Are you trying to get away from them, too? Opinions from the Hangout attendees were mixed and the discussion was illuminating! Let us know below if you’d like to hear more about this topic. ► Subscribe to Our Channel Here: https://bit.ly/2TzgcOu Follow Us Here: Website: https://www.posit.co Hangout: https://pos.it/dsh LinkedIn: https://www.linkedin.com/company/posit-software Bluesky: https://bsky.app/profile/posit.co Thanks for hanging out with us!

Workflow Demo Live Q&A - September 25th!

On September 25th, we hosted a Workflow Demo on data-level permissions using Posit Connect (with Databricks, Snowflake, OAuth: https://youtu.be/ivEoeyWJzVY?feature=shared ) Links mentioned in the Q&A: Release Blurb: https://docs.posit.co/connect/news/#posit-connect-2024.08.0 Security: https://docs.posit.co/connect/admin/integrations/oauth-integrations/security.html Publishing Quarto: https://docs.posit.co/connect/how-to/basic/publish-databricks-quarto-notebook/ sparklyr: https://github.com/sparklyr/sparklyr?tab=readme-ov-file#connecting-through-databricks-connect-v2 odbc: https://github.com/r-dbi/odbc?tab=readme-ov-file#odbc- Helpful resources for this workflow: Full examples to get you started: https://github.com/posit-dev/posit-sdk-py/tree/main/examples/connect Admins will likely be most interested in starting here: https://docs.posit.co/connect/admin/integrations/oauth-integrations/databricks/ End users will be most interested here: https://docs.posit.co/connect/user/oauth-integrations/ Databricks Integrations with Python Cookbook: https://docs.posit.co/connect/cookbook/content/integrations/databricks/python/ Databricks Integrations with R Cookbook: https://docs.posit.co/connect/cookbook/content/integrations/databricks/r/ Snowflake Integrations with Python Cookbook: https://docs.posit.co/connect/cookbook/content/integrations/snowflake/python/

Data-level permissions using Posit Connect (with Databricks, Snowflake, OAuth)

Should one viewer of your app be able to see more (or different) data than another? Maybe colleagues in California should only see data relevant to them? Or managers should only have access to their own employee data? The Connect team joined us for a demo on inheriting data-level permissions using Posit Connect and Databricks Unity Catalog. While this workflow uses Databricks to illustrate federated data access controls, this same methodology can also be applied to Snowflake or any external data source that supports OAuth. During this workflow demo, you will learn: How to define row-level access controls in Databricks Unity Catalog How to create a Databricks OAuth integration in Posit Connect How to write interactive applications that utilize the viewer’s Databricks credentials when reading data from Databricks Unity Catalog, providing the viewer with a personalized experience depending on their level of data access How to deploy this application to Posit Connect and share it within your organization If you’d like to talk further with our team 1:1 about doing this, you can find a time to chat here: https://posit.co/schedule-a-call/?booking_calendar__c=WorkflowDemo Ps. To enable OAuth integrations, your team will need to upgrade to Posit Connect 2024.08.0. This feature is available in Enhanced and Advanced product tiers. Helpful resources for this workflow: Full examples to get you started: https://github.com/posit-dev/posit-sdk-py/tree/main/examples/connect Admins will likely be most interested in starting here: https://docs.posit.co/connect/admin/integrations/oauth-integrations/databricks/ End users will be most interested here: https://docs.posit.co/connect/user/oauth-integrations/ Q&A Link: https://youtube.com/live/TZQY6rm6hU4?feature=share Additional resources shared: Release Blurb: https://docs.posit.co/connect/news/#posit-connect-2024.08.0 Security: https://docs.posit.co/connect/admin/integrations/oauth-integrations/security.html Publishing Quarto: https://docs.posit.co/connect/how-to/basic/publish-databricks-quarto-notebook/ sparklyr: https://github.com/sparklyr/sparklyr?tab=readme-ov-file#connecting-through-databricks-connect-v2 odbc: https://github.com/r-dbi/odbc?tab=readme-ov-file#odbc-

Parallelize R code using user-defined functions in sparklyr

If you’re an Apache Spark user, you benefit from its speed and scalability for big data processing. However, you might still want to leverage R’s extensive ecosystem of packages and intuitive syntax. One effective way to do this is by writing user-defined functions (UDFs) with sparklyr. UDFs enable you to execute R functions within Spark, harnessing Spark’s processing power and combining the strengths of both tools. In this tutorial, you’ll learn how to: Open Posit Workbench as a Databricks user Start a Databricks cluster within Posit Workbench Connect to a cluster within Posit Workbench View Databricks data in RStudio Create a prediction function Create a user-defined function with sparklyr Read our most recent blog that covers parallelizing R code using user-defined functions (UDFs) in sparklyr: https://posit.co/blog/databricks-udfs/ Learn more about our Databricks partnership: https://posit.co/solutions/databricks/ Watch other tutorials on using Databricks and RStudio: https://youtube.com/playlist?list=PL9HYL-VRX0oR-3AgWbXtlfdr29626EjRJ&feature=shared

Predicting Lending Rates with Databricks, tidymodels, and Posit Team

Machine learning algorithms are reshaping financial decision-making, changing how the industry manages financial risk. For our workflow demo today on June 26th at 11 am ET, Garrett Grolemund at Posit will show how to use both Posit and Databricks to apply machine learning methods to the consumer credit market, where accurately predicting lending rates is critical for customer acquisition. *Please note that while the workflow focuses on a financial example, the general workflow will be useful to those using Databricks and R together across any industry. During this workflow demo, you will learn how to: Connect to historical lending rate data stored in Databricks Delta Lake Tune and cross-validate a penalized linear regression (LASSO) that predicts interest rates Select variables with the penalized linear regression model (LASSO) Build an interactive Shiny app to provide a customer-facing user interface for our model Deploy the app to production on Posit Connect, and arrange for the app to access Databricks Resources for the demo: GitHub repo for today’s materials: https://github.com/posit-dev/databricks-finance-app Accompanying Guide: https://pub.demo.posit.team/public/predicting-lending-rates/lending-rate-prediction.html Q&A Recording: https://youtube.com/live/wNI3AhHP7uM Additional follow-up links: GitHub Repo: https://github.com/posit-dev/databricks-finance-app Accompanying Guide: https://pub.demo.posit.team/content/fec42b3d-3aa9-43e1-8312-0ff553d09851/lending-rate-prediction.html While this demo uses ODBC package to connect to Databricks, you can also use sparklyr R package. Learn more about both here: https://docs.posit.co/ide/server-pro/user/rstudio-pro/guide/databricks.html Example using sparklyr instead of ODBC: https://posit.co/blog/reporting-on-nyc-taxi-data-with-rstudio-and-databricks/ Posit Workbench provides additional features for managing Databricks Credentials, learn more here: https://docs.posit.co/ide/server-pro/user/posit-workbench/guide/databricks.html#databricks-with-r For more on the Posit x Databricks partnership: https://posit.co/solutions/databricks/ Blog post on Edgar’s workshop on Databricks at conf: https://posit.co/blog/using-databricks-with-r-conf-workshop/ Solutions article on ODBC and Databricks: https://solutions.posit.co/connections/db/databases/databricks/ Want to chat more with Posit? To talk with Posit about integrating Posit & Databricks: https://posit.co/schedule-a-call/?booking_calendar__c=DatabricksJune2024Demo Had fun and want to join again? You can add the monthly recurring event to your calendar with this link: https://pos.it/team-demo

Connecting RStudio and Databricks with sparklyr

You can connect RStudio and Databricks with the sparklyr package. First, load the necessary packages. Next, set up the connection with sparklyr::spark_connect(). See your data in the Connections pane. Check which databases you’re connected to. Navigate the data structure by expanding the levels. Navigate to the table you want to explore…and analyze your data in RStudio. Learn more: Databricks x Posit: https://posit.co/solutions/databricks/ Empowering R and Python Developers: Databricks and Posit Announce New Integrations: https://posit.co/blog/databricks-and-posit-announce-new-integrations/ Posit Connect: https://posit.co/products/enterprise/connect/ sparklyr and Databricks Connect v2: https://spark.posit.co/deployment/databricks-connect.html Deploying to Posit Connect: https://spark.posit.co/deployment/databricks-posit-connect.html Contact our sales team to schedule a demo: https://posit.co/schedule-a-call/?booking_calendar__c=Databricks

Deploying Databricks-backed content on Posit Connect

After you connect to Databricks using sparklyr and create your content, you’ll want to deploy and share your work. With the new pysparklyr::deploy_databricks() function, you can deploy your content to Posit Connect. Run the function and confirm to start deployment. The deploy_databricks() function will gather your document location, credentials, and URLs. It uses this information to deploy to Posit Connect. After it’s done, click the URL to see the deployed content on Posit Connect. Open the content from the publisher view. Click around and check out your work. In Posit Connect, you can edit sharing permissions, create a vanity URL, and more. Open the content URL to see how your work would look to stakeholders. Learn more: Databricks x Posit: https://posit.co/solutions/databricks/ Empowering R and Python Developers: Databricks and Posit Announce New Integrations: https://posit.co/blog/databricks-and-posit-announce-new-integrations/ Posit Connect: https://posit.co/products/enterprise/connect/ sparklyr and Databricks Connect v2: https://spark.posit.co/deployment/databricks-connect.html Deploying to Posit Connect: https://spark.posit.co/deployment/databricks-posit-connect.html Contact our sales team to schedule a demo: https://posit.co/schedule-a-call/?booking_calendar__c=Databricks

Using R with Databricks Connect - posit::conf(2023)

Presented by Edgar Ruiz Spark Connect, and Databricks Connect, enable the ability to interact with Spark stand-alone clusters remotely. This improves our ability to perform Data Science at-scale. We will share the work in sparklyr, and other products, that will make it easier for R users to take advantage this new framework. Presented at Posit Conference, between Sept 19-20 2023, Learn more at posit.co/conference.# Talk Track: Tidy up your models. Session Code: TALK-1084

Edgar Ruiz

Edgar Ruiz - GitHub Copilot in RStudio

GitHub Copilot in RStudio - Edgar Ruiz Presentation slides available at https://colorado.posit.co/rsc/rstudio-copilot/#/TitleSlide Speaker Bio: Edgar Ruiz is a solutions engineer at Posit with a background in deploying enterprise reporting and business intelligence solutions. He is the author of multiple articles and blog posts sharing analytics insights and server infrastructure for data science. Edgar is the author and administrator of the https://db.rstudio.com web site, and current administrator of the sparklyr web site: https://spark.rstudio.com. Co-author of the dbplyr package, and creator of the dbplot, tidypredict and modeldb package. Presented at the 2023 R/Pharma Conference (October 26, 2023)

Edgar Ruiz

Edgar Ruiz | Programación con R | RStudio (2019)

Hay ocasiones que, cuando trabajamos en un análisis en R, necesitamos dividir nuestros datos en grupos, y después tenemos que correr la misma operación sobre cada grupo. Por ejemplo, puede ser que los datos que tenemos contienen varios países, y necesitamos crear un modelo por cada país. Otro caso sería el de correr múltiples operaciones sobre los mismos datos. Estos casos requieren que sepamos cómo usar iteraciones con R. Este webinar se concentrará en cómo utilizar el paquete llamado purrr para ayudarnos a resolver este tipo de problema. Descargar materiales: https://rstudio.com/resources/webinars/programacio-n-con-r/ About Edgar: Edgar Ruiz es un Ingeniero de Soluciones en RStudio. Es el administrador de los sitios oficiales de sparklyr y de R para bases de datos. También es autor de los paquetes de R: dbplot, tidypredict y modeldb, y co-autor de el paquete dbplyr

Edgar Ruiz

Javier Luraschi | Using pins with Python and JavaScript | RStudio

Last year, pins got released as a brand new R package to pin, discover and cache remote resources for R users. This package has matured to support many use cases; from caching remote URLs, and easily sharing datasets with other R users, to building automated pipelines. However, in order to truly collaborate in multi-disciplinary data-driven teams, one needs to consider how to collaborate beyond R. How can we share resources with designers and machine learning experts who happen to use different programming languages like Python and JavaScript? This talk will introduce the pinsjs project, a cross-language community project which has the goal of bringing pins to the broader open source community to enable rich workflows across larger data-driven teams. About Javier: Javier is the author of “Mastering Spark with R”, pins, sparklyr, mlflow and torch. He holds a double degree in Math and Software Engineer and decades of industry experience with a focus on data analysis. Javier is currently working on a project of his own; and previously worked in RStudio, Microsoft Research and SAP

Edgar Ruiz | Databases using R The latest | RStudio (2019)

Learn about the latest packages and techniques that can help you access and analyze data found inside databases using R. Many of the techniques we will cover are based on our personal and the community’s experiences of implementing concepts introduced last year, such as offloading most of the data wrangling to the database using dplyr, and using the RStudio IDE to preview the database’s layout and data. Also, learn more about the most recent improvements to the RStudio products that are geared to aid developers in using R with databases effectively. VIEW MATERIALS https://github.com/edgararuiz/databases-w-r About the Author Edgar Ruiz Edgar is the author and administrator of the https://db.rstudio.com web site, and current administrator of the [sparklyr] web site: https://spark.rstudio.com. Author of the Data Science in Spark with sparklyr cheatsheet. Co-author of the dbplyr package and creator of the dbplot package

Edgar Ruiz

Javier Luraschi | Scaling R with Spark | RStudio (2019)

This talk introduces new features in sparklyr that enable real-time data processing, brand new modeling extensions and significant performance improvements. The sparklyr package provides an interface to Apache Spark to enable data analysis and modeling in large datsets through familiar packages like dplyr and broom. VIEW MATERIALS https://github.com/rstudio/rstudio-conf/tree/master/2019/Scaling%20R%20with%20Spark%20-%20Javier%20Luraschi About the Author Javier Luraschi Javier is a Software Engineer with experience in technologies ranging from desktop, web, mobile and backend; to augmented reality and deep learning applications. He previously worked for Microsoft Research and SAP and holds a double degree in Mathematics and Software Engineering

Extending Spark Using Sparklyr | RStudio Webinar - 2017

This is a recording of an RStudio webinar. You can subscribe to receive invitations to future webinars at https://www.rstudio.com/resources/webinars/. We try to host a couple each month with the goal of furthering the R community’s understanding of R and RStudio’s capabilities. We are always interested in receiving feedback, so please don’t hesitate to comment or reach out with a personal message

Sparklyr: Using Spark with RMarkdown | RStudio Webinar - 2016

This is a recording of an RStudio webinar. You can subscribe to receive invitations to future webinars at https://www.rstudio.com/resources/web …. We try to host a couple each month with the goal of furthering the R community’s understanding of R and RStudio’s capabilities. We are always interested in receiving feedback, so please don’t hesitate to comment or reach out with a personal message

Understanding Sparklyr Deployment Modes | RStudio Webinar - 2017

This is a recording of an RStudio webinar. You can subscribe to receive invitations to future webinars at https://www.rstudio.com/resources/webinars/. We try to host a couple each month with the goal of furthering the R community’s understanding of R and RStudio’s capabilities. We are always interested in receiving feedback, so please don’t hesitate to comment or reach out with a personal message

SparklyR