We've recently open-sourced another tool, called D-Tale (a.k.a. ‘the details of your data’).
D-Tale is a lightweight tool that allows users to view all aspects of their data, while maintaining the ability to perform simple operations such as sorting, filtering and formatting.
D-Tale was born out of a pre-existing internal solution for viewing SAS datasets, but meant for users of Python Pandas data structures, e.g. Series and DataFrames.
This tool is strongly geared towards anyone who used Pandas data structures (Series, DataFrame etc) in their day-to-day operations and need an easier way of visualising data without writing cumbersome Python in an editor (which is still the way to go if you need full control).
As well as displaying data in a performant web-based UI, D-Tale also supports:
- Basic Grid Operations (Sorting, Filtering, Column Locking)
- Coverage Analysis
- Correlation Matrices (with the ability to drill into each individual correlation in a timeseries or single-day format)
- With many other features in development...
The tool supports both Python 2 & 3 as well as Docker integrations. Our solution stores your entire piece of data in the memory of a Python Flask process and then renders a small subset of rows / columns based on the dimensions of your browser using react-virtualized. We found this method to be the most performant while at the same time giving the user the impression that all of their data is being rendered on the screen.
Here are instructions to get started:
- Setup/Activate your environment and install the egg
# create a virtualenv, if you haven't already created one (use "python -m virtualenv ~/pyenvs/dtale" if you're running Python2)
$ python3 -m venv ~/pyenvs/dtale
$ source ~/pyenvs/dtale/bin/activate
# install dtale egg (important to use the "-U" every time you install so it will grab the latest version)
$ pip install --upgrade dtale
- Now you will have to ability to use D-Tale from the command-line or within a Python-enabled terminal
- Loading data from arctic:
dtale --arctic-host mongodb://localhost:27027 --arctic-library jdoe.my_lib --arctic-node my_node --arctic-start 20130101 --arctic-end 20161231
- Loading data from CSV:
dtale --csv-path /home/jdoe/my_csv.csv --csv-parse_dates date
- Loading data from arctic:
- Python-enabled terminal
dtale.show(df) # FYI: df is a Pandas dataframe
Prospective users should look to the GitHub Readme for set up instructions, working examples and further _d-tales_ of the features we offer.
We're committed to contributing back to the open-source community upon which we heavily rely. We've recently open-sourced D-Tale through our GitHub Account in the hope other users will see the same time-saving benefits we've benefitted from. We're always keen to improve though so pull requests and issues against the repo are very welcome!