{"id":1604,"date":"2024-05-30T17:06:10","date_gmt":"2024-05-30T15:06:10","guid":{"rendered":"https:\/\/easybi.lu\/?p=1604"},"modified":"2025-01-10T08:33:33","modified_gmt":"2025-01-10T07:33:33","slug":"dataframes-with-pandas-on-jupyterlab","status":"publish","type":"post","link":"https:\/\/easybi.lu\/index.php\/2024\/05\/30\/dataframes-with-pandas-on-jupyterlab\/","title":{"rendered":"Dataframes with Pandas on JupyterLab"},"content":{"rendered":"\n

Introduction<\/h2>\n\n\n\n

Pandas standing for panel data<\/a> is managing an outstanding object called DataFrame <\/strong>and the first time I encountered the famous \u00ab\u00a0dataframe object<\/a>\u00a0\u00bb concept was when I was asked to put in place an Apache Spark SQL environment in my current customer’s day.<\/p>\n\n\n\n

As a SQL and java developper, I didn’t immediatly realize that I ran into one of the most powerful API to manage data, a mix between SQL and functional programming using lambda functions.<\/p>\n\n\n\n

The dataframe (or dataset of rows, thanks to java \ud83d\ude00 ) is technicaly speaking a simple table and can be manipulated in ease with functions inspired from SQL keywords, this brillant idea leads to the consecration of Apache Spark version 2 with its Spark SQL module<\/a>, but that not the subject of this blog post \ud83d\ude09<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

<\/p>\n\n\n\n

Pandas VS Pandas on Apache Spark<\/h2>\n\n\n\n

As a new python developper I recently discover Pandas and this is a really better starting point to learn the concept than installing a complete Apache Spark instance, you don’t need to know about technical background concept like ressource management<\/a>, DAG<\/a>, Catalyst engine optimizer, clustered machine<\/a> principle, etc… just install the python libraries and we are good to go and even better, we can use a notebook editor like JupyterLab<\/a> using another simple pip install command.<\/p>\n\n\n\n

As you can see the complete difference here<\/a> between classic Pandas intallation and Pandas on Spark, Pandas is easy to familiarize yourself with, as its single-threaded and just simply put the data in memory, very convenient to quickly manage a csv or excel dataset.<\/p>\n\n\n\n

To go further, check also the full pyspark getting started<\/a>, pay attention that there are some limitations due to distributed and parallel execution model, I would suggest to use the pure Spark API<\/a>.<\/p>\n\n\n\n

Demo<\/h2>\n\n\n\n

Ok, talking is cheap let’s see how it works, I just introduce two simple ways to open a notebook and launch a code sample<\/p>\n\n\n\n