Jekyll2023-12-31T02:03:35-08:00https://thedebugger811.github.io/feed.xmlParth Kothari Home PageThird Year PhD StudentParth Kothariparthkothari811@gmail.comEmpirical Evidence2023-12-31T00:00:00-08:002023-12-31T00:00:00-08:00https://thedebugger811.github.io/posts/2023/12/EmpiricalEvidence<p>‘Do you think it’s possible to get six packs by just exercising daily and controlling the food intake? I mean without taking extra whey protein and supplements and without going to the gym.’</p>
<p>‘I haven’t seen any empirical evidence.’</p>
<p>‘Lets verify it then.’</p>
<p><a href="https://docs.google.com/spreadsheets/d/13oA2G1ytE0PrlelrBYbThullfPLUK75wBBxjHNt3qv0/edit?usp=sharing">Google Sheet</a></p>Parth Kothariparthkothari811@gmail.com‘Do you think it’s possible to get six packs by just exercising daily and controlling the food intake? I mean without taking extra whey protein and supplements and without going to the gym.’Deep Learning for Autonomous Vehicles: Milestone 12021-04-14T00:00:00-07:002021-04-14T00:00:00-07:00https://thedebugger811.github.io/posts/2021/04/DLAV<p>In this post, I provide a kickstarter guide to getting started with TrajNet++ framework for human trajectory forecasting, which will prove useful in helping you approach Milestone 1.</p>
<h4 id="updates-22042021">Updates (22.04.2021)</h4>
<ol>
<li>Added an FAQ section.</li>
<li>Updated Visualization section.</li>
</ol>
<h4 id="updates-20042021">Updates (20.04.2021)</h4>
<ol>
<li>Resources to get started with GitHub added.</li>
</ol>
<h4 id="updates-19042021">Updates (19.04.2021)</h4>
<ol>
<li>Procedure to setting up TrajNet++ on SCITAS added.</li>
</ol>
<h4 id="updates-16042021">Updates (16.04.2021)</h4>
<ol>
<li>Procedure to submit to AICrowd added.</li>
</ol>
<h4 id="updates-15042021">Updates (15.04.2021)</h4>
<ol>
<li>Git fork the code, and then clone the <em>forked repository</em>. This will help the TAs to track your code easily.</li>
<li>The latest push on trajnetplusplusbaselines now works with Python 3.8</li>
<li>The Next Steps of Milestone 1 have been added, at the end.</li>
</ol>
<h6 id="working-with-15042021">Working with (15.04.2021)</h6>
<ol>
<li>MacOS 10.15.7 + Conda (Python3.6 & Python3.8) ✅</li>
<li>Ubuntu 18.04 + Virtualenv (Python3.6 & Python3.8) ✅</li>
</ol>
<h1 id="overview">Overview</h1>
<p>On a high-level, Trajnet++ constitutes four primary components:</p>
<ol>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplustools">Trajnetplusplustools</a>: This repository provides helper functions for trajectory prediction. For instance: trajectory categorization, evaluation metrics, prediction visualization.</p>
</li>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplusdataset">Trajnetplusplusdataset</a>: This repository provides scripts to generate train, val and test dataset splits from raw data as well as simulators.</p>
</li>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplusbaselines">Trajnetplusplusbaselines</a>: This repository provides baseline models (handcrafted as well as data-driven) for human motion prediction. This repository also provides scripts to extensively evaluate the trained models.</p>
</li>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplusdata">Trajnetplusplusdata</a>: This repository provides the already processed real world data as well as synthetic datasets conforming to human motion.</p>
</li>
</ol>
<h1 id="milestone-1-getting-started">Milestone 1: Getting Started</h1>
<p>I describe on how to get started using TrajNet++ with the help of a running example.
We will download an already-created synthetic dataset and train an LSTM-based model to perform trajectory forecasting.</p>
<h2 id="setting-up-the-repository">Setting Up The Repository</h2>
<p>The first step is to setup the repository Trajnetplusplusbaselines for model training. Next, we setup the virtual environment and download the requirements. The virtual environment can also be setup using <a href="https://docs.conda.io/projects/conda/en/latest/user-guide/cheatsheet.html">Conda</a> on local machines.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="c">## 1. On LOCAL MACHINE</span>
<span class="c">## Make virtual environment using either A. virtualenv or B. conda</span>
<span class="c">## A. Using virtualenv</span>
<span class="c">## Works with Python3.6 and Python3.8</span>
virtualenv <span class="nt">-p</span> /usr/bin/python3.6 trajnetv
<span class="nb">source </span>trajnetv/bin/activate
<span class="c">## B. Using conda</span>
<span class="c">## Works with Python3.6 and Python3.8</span>
conda create <span class="nt">--name</span> trajnetv <span class="nv">python</span><span class="o">=</span>3.8
conda activate trajnetv
<span class="c">## 2. On SCITAS</span>
module load gcc
module load python/3.7.3
virtualenv <span class="nt">--system-site-packages</span> venvs/trajnetv
<span class="nb">source </span>venvs/trajnetv/bin/activate
</code></pre></div></div>
<p>Set up TrajNet++ on SCITAS after verifying the setup on local machine. For SCITAS, no need to fork the repository again. Clone the already-created forked repository.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Create directory to setup Trajnet++</span>
<span class="nb">mkdir </span>trajnet++
<span class="nb">cd </span>trajnet++
<span class="c">## Clone Repositories</span>
<span class="c"># git clone https://github.com/vita-epfl/trajnetplusplusbaselines.git (Old)</span>
git clone <forked_repository>
<span class="c">## Download Requirements</span>
<span class="nb">cd </span>trajnetplusplusbaselines/
pip <span class="nb">install</span> <span class="nt">-e</span> <span class="nb">.</span>
<span class="c">## SCITAS-Specific (!)</span>
<span class="c">## If previous command gives an error: "requires deeptoolsintervals>=0.1.7, requires plotly>=2.0.0", then:</span>
pip <span class="nb">install </span>deeptoolsintervals
pip <span class="nb">install </span>plotly
pip <span class="nb">install</span> <span class="nt">-e</span> <span class="nb">.</span>
</code></pre></div></div>
<p>Follow the next steps for SCITAS as well.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Additional Requirements (ORCA)</span>
wget https://github.com/sybrenstuvel/Python-RVO2/archive/master.zip
unzip master.zip
<span class="nb">rm </span>master.zip
<span class="c">## Setting up ORCA (steps provided in the Python-RVO2 repo)</span>
<span class="nb">cd </span>Python-RVO2-master/
pip <span class="nb">install </span>cmake
pip <span class="nb">install </span>cython
python setup.py build
python setup.py <span class="nb">install
cd</span> ../
</code></pre></div></div>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Additional Requirements (Social Force)</span>
wget https://github.com/svenkreiss/socialforce/archive/refs/heads/main.zip
unzip main.zip
<span class="nb">rm </span>main.zip
<span class="c">## Setting up Social Force</span>
<span class="nb">cd </span>socialforce-main/
pip <span class="nb">install</span> <span class="nt">-e</span> <span class="nb">.</span>
<span class="nb">cd</span> ../
</code></pre></div></div>
<p>Our repository is now setup!</p>
<h2 id="preparing-the-dataset">Preparing the Dataset</h2>
<p>Now, we will download and prepare data for training our models. In this example, we download a synthetic dataset generated using ORCA policy.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd </span>DATA_BLOCK/
wget https://github.com/vita-epfl/trajnetplusplusdata/releases/download/v3.1/five_parallel_synth.zip
unzip five_parallel_synth.zip
<span class="nb">rm </span>five_parallel_synth.zip
<span class="nb">ls </span>five_parallel_synth/
</code></pre></div></div>
<p>You will notice that the current folder contains <em>train</em> data, <em>test</em> data and <em>test_private</em> data. The <em>test</em> data contains the test examples uptil the end of observation period, while the <em>test_private</em>, as the name suggests, contains the ground-truth predictions which will be used as reference to evaluate the performing of forecasting model. You will notice that a <em>validation</em> set is not present. Preparing a validation set is important for performing hyperparameter tuning. You can use the following helper file to split the current training samples into training (80%) and validation split (20%).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">cd</span> ../
python create_validation.py <span class="nt">--path</span> five_parallel_synth <span class="nt">--val_ratio</span> 0.2
</code></pre></div></div>
<p>The above command will create a new folder <em>five_parallel_synth_split</em> in the <em>DATA_BLOCK</em> folder.</p>
<p>We can additionally transfer the goal information of the dataset in the <em>goal_files</em> folder.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Preparing Goals folder (additional attributes)</span>
<span class="nb">mkdir</span> <span class="nt">-p</span> goal_files/train
<span class="nb">mkdir </span>goal_files/val
<span class="nb">mkdir </span>goal_files/test_private
<span class="c">## For other datasets, the goal files can be different for corresponding dataset split</span>
<span class="nb">cp </span>DATA_BLOCK/five_parallel_synth/orca_five_nontraj_synth.pkl goal_files/train/
<span class="nb">cp </span>DATA_BLOCK/five_parallel_synth/orca_five_nontraj_synth.pkl goal_files/val/
<span class="nb">cp </span>DATA_BLOCK/five_parallel_synth/orca_five_nontraj_synth.pkl goal_files/test_private/
</code></pre></div></div>
<p>Now that the dataset is ready, its time to train the model! :)</p>
<h2 id="training-models">Training Models</h2>
<p>Training models is more easier than setting up Trajnet++ !</p>
<p>For SCITAS, the training takes place using bash-scripts, please refer to the tutorial to understand more. The below training procedure is only for your local machines.</p>
<p>All you got to do is ….</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">trainer</span> <span class="o">--</span><span class="n">path</span> <span class="n">five_parallel_synth_split</span> <span class="o">--</span><span class="n">augment</span>
</code></pre></div></div>
<p>…. and your LSTM model starts training. Your model will be saved in the <em>five_parallel_synth_split</em> folder within <em>OUTPUT_BLOCK</em>. Currently, models are being saved according to the type of interaction models being used.</p>
<p>In order to train using interaction modules (eg. directional-grid) utilizing additional attribute (goal information), run</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">trainer</span> <span class="o">--</span><span class="n">path</span> <span class="n">five_parallel_synth_split</span> <span class="o">--</span><span class="nb">type</span> <span class="s">'directional'</span> <span class="o">--</span><span class="n">goals</span> <span class="o">--</span><span class="n">augment</span>
</code></pre></div></div>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## To know more options about trainer
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">trainer</span> <span class="o">--</span><span class="n">help</span>
</code></pre></div></div>
<p>For models trained on SCITAS, you can evaluate and visualize these models on your local machine. To do so, ‘scp’ the output files and log files from SCITAS to the repository on your local machine. Note: Maintain the same file structure.</p>
<h2 id="evaluating-models">Evaluating Models</h2>
<p>One strength of TrajNet++ is its extensive evaluation system. You can read more about it in the <a href="https://www.aicrowd.com/challenges/trajnet-a-trajectory-forecasting-challenge">metrics section here</a>.</p>
<p>To perform extensive evaluation of your trained model. The results are saved in Results.png</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># python -m evaluator.trajnet_evaluator --path <test_dataset> --output <path_to_model_pkl_file>
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">trajnet_evaluator</span> <span class="o">--</span><span class="n">path</span> <span class="n">five_parallel_synth</span> <span class="o">--</span><span class="n">output</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth_split</span><span class="o">/</span><span class="n">lstm_vanilla_None</span><span class="p">.</span><span class="n">pkl</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth_split</span><span class="o">/</span><span class="n">lstm_goals_directional_None</span><span class="p">.</span><span class="n">pkl</span>
<span class="c1">## To know more options about evaluator
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">trajnet_evaluator</span> <span class="o">--</span><span class="n">help</span>
</code></pre></div></div>
<p>To know more about how the evaluation procedure works, please refer to this <a href="https://github.com/vita-epfl/trajnetplusplusbaselines/blob/master/evaluator/README.rst">README</a>.</p>
<h2 id="visualize-models">Visualize Models</h2>
<p>Visualize learning curves of two different models</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">plot_log</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth_split</span><span class="o">/</span><span class="n">lstm_vanilla_None</span><span class="p">.</span><span class="n">pkl</span><span class="p">.</span><span class="n">log</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth_split</span><span class="o">/</span><span class="n">lstm_goals_directional_None</span><span class="p">.</span><span class="n">pkl</span><span class="p">.</span><span class="n">log</span>
<span class="c1">## To view the different log files generated, run the command below:
</span><span class="n">ls</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth_split</span><span class="o">/</span><span class="n">lstm_goals_directional_None</span><span class="p">.</span><span class="n">pkl</span><span class="o">*</span>
<span class="c1">## You will notice various log files in form of *.png
</span></code></pre></div></div>
<p>Visualize predictions of models</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># python -m evaluator.visualize_predictions <ground_truth_file> <prediction_files>
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">visualize_predictions</span> <span class="n">DATA_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth</span><span class="o">/</span><span class="n">test_private</span><span class="o">/</span><span class="n">orca_five_nontraj_synth</span><span class="p">.</span><span class="n">ndjson</span> <span class="n">DATA_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth</span><span class="o">/</span><span class="n">test_pred</span><span class="o">/</span><span class="n">lstm_vanilla_None_modes1</span><span class="o">/</span><span class="n">orca_five_nontraj_synth</span><span class="p">.</span><span class="n">ndjson</span> <span class="n">DATA_BLOCK</span><span class="o">/</span><span class="n">five_parallel_synth</span><span class="o">/</span><span class="n">test_pred</span><span class="o">/</span><span class="n">lstm_goals_directional_None_modes1</span><span class="o">/</span><span class="n">orca_five_nontraj_synth</span><span class="p">.</span><span class="n">ndjson</span> <span class="o">--</span><span class="n">labels</span> <span class="n">Vanilla</span> <span class="n">D</span><span class="o">-</span><span class="n">Grid</span> <span class="o">--</span><span class="n">n</span> <span class="mi">10</span> <span class="o">--</span><span class="n">random</span> <span class="o">-</span><span class="n">o</span> <span class="n">visualize</span>
<span class="c1">## Note the addition of output argument above. The 10 random predictions are saved in the trajnetplusplusbaselines directory. Run:
</span><span class="n">ls</span> <span class="n">visualize</span><span class="o">*</span><span class="p">.</span><span class="n">png</span>
<span class="c1">## You wil see 10 '.png' files with prefix 'visualize' as it was the provided output name.
</span></code></pre></div></div>
<h2 id="next-steps-for-milestone-1">Next Steps for Milestone 1</h2>
<ol>
<li>
<p>Add visualizations (obtained using previous command) of 3 test scenes qualitatively comparing outputs of the vanilla model and D-Grid model (that uses goal information), as well as the quantitative evaluation (Results.png) in the README file of your forked repository.</p>
</li>
<li>Next, train the vanilla model and the D-Grid model on TrajNet++ synthetic data and real data following the same steps as above. Link to <a href="https://github.com/vita-epfl/trajnetplusplusdata/releases/download/v4.0/train.zip">Train data</a> and <a href="https://github.com/vita-epfl/trajnetplusplusdata/releases/download/v4.0/test.zip">Test data</a>. Note that, you will need to make different folders for <em>real_data</em> and <em>synth_data</em> in <em>DATA_BLOCK</em> folder. Your TrajNet++ data folders should have structure similar to below:
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DATA_BLOCK
│
└───real_data
│ └── train
| └── val (self-generated)
│ └── test
│ │ crowds_zara02.ndjson
│ │ biwi_eth.ndjson
│ │ crowds_uni_examples.ndjson
│
└───synth_data
│ └── train
| └── val (self-generated)
│ └── test
│ │ orca_synth.ndjson
| | collision_test.ndjson
</code></pre></div> </div>
</li>
<li>For faster training on real data, you can remove the CFF files the <em>train</em> folder.
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">rm</span> <span class="o"><</span><span class="n">path_to_real_data</span><span class="o">>/</span><span class="n">train</span><span class="o">/</span><span class="n">cff</span><span class="o">*</span>
</code></pre></div> </div>
</li>
<li>
<p>You are encouraged to play with other interaction encoders and maybe, design your own! You can validate your designs on the synthetic data before trying out on real data.</p>
</li>
<li>Final Step: Upload the predictions of D-Grid on <a href="https://www.aicrowd.com/challenges/trajnet-a-trajectory-forecasting-challenge">AICrowd</a>. You will have to create an account on AICrowd as well as accept the terms and conditions for the TrajNet++ challenge.</li>
</ol>
<h2 id="submission-to-aicrowd">Submission to AICrowd</h2>
<p>Lets assume you have two models named ‘synth_model_name’ trained on TrajNet++ synthetic data and ‘real_model_name’ trained on TrajNet++ real data. Also, by default, you have the data directory structure as mentioned above.</p>
<h3 id="generating-predictions-for-aicrowd">Generating Predictions for AICrowd</h3>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## Generate for Real data
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">trajnet_evaluator</span> <span class="o">--</span><span class="n">path</span> <span class="n">real_data</span> <span class="o">--</span><span class="n">output</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">real_data</span><span class="o">/<</span><span class="n">real_model_name</span><span class="o">></span><span class="p">.</span><span class="n">pkl</span> <span class="o">--</span><span class="n">write_only</span>
<span class="c1">## Generate for Real data
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">trajnet_evaluator</span> <span class="o">--</span><span class="n">path</span> <span class="n">synth_data</span> <span class="o">--</span><span class="n">output</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">synth_data</span><span class="o">/<</span><span class="n">synth_model_name</span><span class="o">></span><span class="p">.</span><span class="n">pkl</span> <span class="o">--</span><span class="n">write_only</span>
</code></pre></div></div>
<p>The above operations will save your model predictions in the <em>test_pred</em> folder within data directory as shown below:</p>
<div class="language-plaintext highlighter-rouge"><div class="highlight"><pre class="highlight"><code>DATA_BLOCK
│
└───real_data
│ └── train
| └── val (self-generated)
│ └── test_pred
| └── <real_model_name>_modes1
│ | crowds_zara02.ndjson
│ | biwi_eth.ndjson
│ | crowds_uni_examples.ndjson
│ └── test
└───synth_data
│ └── train
| └── val (self-generated)
│ └── test_pred
| └── <synth_model_name>_modes1
│ | orca_synth.ndjson
| | collision_test.ndjson
│ └── test
</code></pre></div></div>
<h3 id="uploading-predictions-to-aicrowd">Uploading Predictions to AICrowd</h3>
<p>These test predictions need to be uploaded to AICrowd.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## KEEP THE SAME FOLDER NAMES and STRUCTURE given below !!
</span><span class="n">mkdir</span> <span class="n">test</span>
<span class="n">mkdir</span> <span class="n">test</span><span class="o">/</span><span class="n">real_data</span>
<span class="n">mkdir</span> <span class="n">test</span><span class="o">/</span><span class="n">synth_data</span>
<span class="n">cp</span> <span class="n">DATA_BLOCK</span><span class="o">/</span><span class="n">real_data</span><span class="o">/</span><span class="n">test_pred</span><span class="o">/<</span><span class="n">real_model_name</span><span class="o">></span><span class="n">_modes1</span><span class="o">/*</span> <span class="n">test</span><span class="o">/</span><span class="n">real_data</span>
<span class="n">cp</span> <span class="n">DATA_BLOCK</span><span class="o">/</span><span class="n">synth_data</span><span class="o">/</span><span class="n">test_pred</span><span class="o">/<</span><span class="n">synth_model_name</span><span class="o">></span><span class="n">_modes1</span><span class="o">/*</span> <span class="n">test</span><span class="o">/</span><span class="n">synth_data</span>
<span class="nb">zip</span> <span class="o">-</span><span class="n">r</span> <span class="o"><</span><span class="n">my_model_name</span><span class="o">></span><span class="p">.</span><span class="nb">zip</span> <span class="n">test</span><span class="o">/</span>
<span class="c1">## Upload the <my_model_name>.zip to AICrowd.
</span></code></pre></div></div>
<p>Done Done! :)</p>
<h1 id="useful-resources">Useful Resources</h1>
<h3 id="introduction-to-git">Introduction to Git</h3>
<p>To help you get started with Git, here are some useful resources:</p>
<p>Git Handbook (10 min. read): <a href="https://guides.github.com/introduction/git-handbook/">https://guides.github.com/introduction/git-handbook/</a></p>
<p>Git Cheatsheet: <a href="https://training.github.com/downloads/github-git-cheat-sheet/">https://training.github.com/downloads/github-git-cheat-sheet/</a></p>
<h3 id="faq">FAQ</h3>
<h6 id="q1-important-steps-when-you-come-back-to-the-code">Q1. Important steps when you come back to the code</h6>
<p>Please activate your virtual environment!</p>
<h6 id="q2-important-steps-before-you-close-your-code-after-a-good-days-progress-">Q2. Important steps before you close your code after a good day’s progress :)</h6>
<p>Do not forget to <strong>push</strong> your code on GitHub. It saves your progress! :) Refer to the GitHub resources if you haven’t yet.</p>
<h6 id="q3-what-are-the-goal-files-in-synthetic-data-why-are-they-absent-in-real-data">Q3. What are the ‘goal’ files in synthetic data, why are they absent in real data?</h6>
<p>The goal files contains the ‘final destination’ (goal) of the pedestrian in the ORCA simulator. It does not refer to the location at the end of the prediction period, but the location at the end of the simulation. We have access to these goals only for synthetic data, so only use ‘–goals’ command for synthetic data and <strong>not</strong> real data. Remember to shift the goal .pkl file to the <em>goal_files</em> folder as shown in the tutorial above.</p>
<h6 id="q4-how-do-we-open-the-png-files-which-are-generated-when-we-run-the-visualization-command">Q4. How do we open the .png files which are generated when we run the visualization command?</h6>
<p>You can transfer the .png files using ‘scp’ to your Desktop (the boring but simple way). Or you can use a text editor that allows you to open .png files from the terminal. I use <a href="https://www.sublimetext.com/">Sublime Text</a>.</p>Parth Kothariparthkothari811@gmail.comIn this post, I provide a kickstarter guide to getting started with TrajNet++ framework for human trajectory forecasting, which will prove useful in helping you approach Milestone 1.TrajNet++ : Dataset Conversion2020-10-03T00:00:00-07:002020-10-03T00:00:00-07:00https://thedebugger811.github.io/posts/2020/10/DatasetConversion<p>In this blog post, I provide a quick tutorial to converting external datasets into the desired <a href="http://ndjson.org/">.ndjson</a> format using the TrajNet++ framework. This post will focus on utilizing the TrajNet++ dataset code for easily converting new datasets.</p>
<!-- Overview
========
On a high-level, Trajnet++ dataset code does three primary functions:
1. [readers.py](https://github.com/vita-epfl/trajnetplusplusdataset/blob/master/trajnetdataset/readers.py): This code converts the raw dataset files into trackrows in .ndjson format
2. [scene.py](https://github.com/vita-epfl/trajnetplusplusdataset/blob/master/trajnetdataset/scene.py): This code constructs the different scenes (i.e. the dataset samples), as per the specification provided, from the trackrows.
3. [get_type.py](https://github.com/vita-epfl/trajnetplusplusdataset/blob/master/trajnetdataset/get_type.py): This code performs trajectory categorization for every scene. Read more about our trajectory categorization [here](https://arxiv.org/pdf/2007.03639v2.pdf). -->
<h1 id="details">Details</h1>
<p>In this tutorial, I will convert the ETH dataset utilized by the <a href="https://github.com/agrimgupta92/sgan">Social GAN</a> paper.</p>
<h2 id="step-1-downloading-data">Step 1: Downloading Data</h2>
<p>Before proceeding, please setup the base repositories. See ‘Setting Up Repositories’ <a href="https://thedebugger811.github.io/posts/2020/03/intro_trajnetpp/">here</a></p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Checkout 'eth' branch of Trajnetplusplusdataset</span>
git checkout <span class="nt">-b</span> eth origin/eth
<span class="c">## Download external data</span>
sh download_data.sh
<span class="nb">cp</span> <span class="nt">-r</span> datasets/eth/ data/
</code></pre></div></div>
<h2 id="step-2-converting-raw-data">Step 2: Converting Raw Data</h2>
<ol>
<li>In our external dataset, each trajectory point is delimited by ‘\t’</li>
<li>TrackRow takes the arguments ‘frame’, ‘ped_id’, ‘x’, ‘y’ in order.</li>
</ol>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">standard</span><span class="p">(</span><span class="n">line</span><span class="p">):</span>
<span class="n">line</span> <span class="o">=</span> <span class="p">[</span><span class="n">e</span> <span class="k">for</span> <span class="n">e</span> <span class="ow">in</span> <span class="n">line</span><span class="p">.</span><span class="n">split</span><span class="p">(</span><span class="s">'</span><span class="se">\t</span><span class="s">'</span><span class="p">)</span> <span class="k">if</span> <span class="n">e</span> <span class="o">!=</span> <span class="s">''</span><span class="p">]</span>
<span class="k">return</span> <span class="n">TrackRow</span><span class="p">(</span><span class="nb">int</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">line</span><span class="p">[</span><span class="mi">0</span><span class="p">])),</span>
<span class="nb">int</span><span class="p">(</span><span class="nb">float</span><span class="p">(</span><span class="n">line</span><span class="p">[</span><span class="mi">1</span><span class="p">])),</span>
<span class="nb">float</span><span class="p">(</span><span class="n">line</span><span class="p">[</span><span class="mi">2</span><span class="p">]),</span>
<span class="nb">float</span><span class="p">(</span><span class="n">line</span><span class="p">[</span><span class="mi">3</span><span class="p">]))</span>
</code></pre></div></div>
<p>Code snippet already provided in <a href="https://github.com/vita-epfl/trajnetplusplusdataset/blob/eth/trajnetdataset/readers.py">readers.py</a></p>
<h2 id="dataset-generation-and-categorization">Dataset Generation and Categorization</h2>
<p>For dataset conversion, we call the ‘raw dataset conversion’ code shown above in convert.py</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">standard</span><span class="p">(</span><span class="n">sc</span><span class="p">,</span> <span class="n">input_file</span><span class="p">):</span>
<span class="k">print</span><span class="p">(</span><span class="s">'processing '</span> <span class="o">+</span> <span class="n">input_file</span><span class="p">)</span>
<span class="k">return</span> <span class="p">(</span><span class="n">sc</span>
<span class="p">.</span><span class="n">textFile</span><span class="p">(</span><span class="n">input_file</span><span class="p">)</span>
<span class="p">.</span><span class="nb">map</span><span class="p">(</span><span class="n">readers</span><span class="p">.</span><span class="n">standard</span><span class="p">)</span>
<span class="p">.</span><span class="n">cache</span><span class="p">())</span>
</code></pre></div></div>
<p>Code snippet already provided in convert.py</p>
<p>Finally, we call the appropriate data files for conversion and categorization (See <a href="https://github.com/vita-epfl/trajnetplusplusdataset/blob/eth/trajnetdataset/convert.py">convert.py</a>).</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>python <span class="nt">-m</span> trajnetdataset.convert <span class="nt">--obs_len</span> 8 <span class="nt">--pred_len</span> 12
</code></pre></div></div>
<p>Now that the dataset is ready [in <em>output</em> folder], you can train the model! :)</p>
<h1 id="difference-in-generated-data">Difference in generated data</h1>
<ol>
<li>Partial tracks are now included (for correct occupancy maps)</li>
<li>Pedestrians that appear in multiple chunks had the same id before (might be a problem for some input readers)</li>
<li>Explicit index of scenes with annotation of the primary pedestrian</li>
</ol>
<h1 id="summarizing">Summarizing</h1>
<p>So, for converting any external dataset, all you got to do is 4 simple steps:</p>
<ol>
<li>Download data and place it in the /data folder.</li>
<li>Edit <a href="https://github.com/vita-epfl/trajnetplusplusdataset/blob/master/trajnetdataset/readers.py">readers.py</a> to convert raw format into TrackRows in <a href="http://ndjson.org/">.ndjson</a> format</li>
<li>Call the above snippet in <a href="https://github.com/vita-epfl/trajnetplusplusdataset/blob/master/trajnetdataset/convert.py">convert.py</a></li>
<li>Call the dataset generation code with the appropriate arguments.</li>
</ol>
<p>We recently released TrajNet++ Challenge for agent-agent based trajectory forecasting. Details regarding the challenge can be found <a href="https://www.aicrowd.com/challenges/trajnet-a-trajectory-forecasting-challenge">here</a>.</p>Parth Kothariparthkothari811@gmail.comIn this blog post, I provide a quick tutorial to converting external datasets into the desired .ndjson format using the TrajNet++ framework. This post will focus on utilizing the TrajNet++ dataset code for easily converting new datasets.Introducing TrajNet++ : A Framework for Human Trajectory Forecasting2020-03-27T00:00:00-07:002020-03-27T00:00:00-07:00https://thedebugger811.github.io/posts/2020/03/IntroducingTrajnetpp<p>In this blog post, I provide a kickstarter guide to our recently released TrajNet++ framework for human trajectory forecasting. We recently released TrajNet++ Challenge for agent-agent based trajectory forecasting. Details regarding the challenge can be found <a href="https://www.aicrowd.com/challenges/trajnet-a-trajectory-forecasting-challenge">here</a>. This post will focus on utilizing the TrajNet++ framework for easily creating datasets and learning human motion forecasting models.</p>
<h1 id="overview">Overview</h1>
<p>On a high-level, Trajnet++ constitutes four primary components:</p>
<ol>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplustools">Trajnetplusplustools</a>: This repository provides helper functions for trajectory prediction. For instance: trajectory categorization, evaluation metrics, prediction visualization.</p>
</li>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplusdataset">Trajnetplusplusdataset</a>: This repository provides scripts to generate train, val and test dataset splits from raw data as well as simulators.</p>
</li>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplusbaselines">Trajnetplusplusbaselines</a>: This repository provides baseline models (handcrafted as well as data-driven) for human motion prediction. This repository also provides scripts to extensively evaluate the trained models.</p>
</li>
<li>
<p><a href="https://github.com/vita-epfl/trajnetplusplusdata">Trajnetplusplusdata</a>: This repository provides the already processed real world data as well as synthetic datasets conforming to human motion.</p>
</li>
</ol>
<h1 id="getting-started">Getting Started</h1>
<p>I describe how to get started using TrajNet++ with the help of a running example.
We will create a synthetic dataset using ORCA simulator and train an LSTM-based model to perform trajectory prediction.</p>
<h2 id="setting-up-repositories">Setting Up Repositories</h2>
<p>The first step is to setup the repositories, namely Trajnetplusplusdata for dataset generation and Trajnetplusplusbaselines for model training. Next, we setup the virtual environment and download the requirements.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Create directory to setup Trajnet++</span>
<span class="nb">mkdir </span>trajnet++
<span class="nb">cd </span>trajnet++
<span class="c">## Clone Repositories</span>
git clone https://github.com/vita-epfl/trajnetplusplusdataset.git
git clone https://github.com/vita-epfl/trajnetplusplusbaselines.git
<span class="c">## Make virtual environment</span>
virtualenv <span class="nt">-p</span> /usr/bin/python3.6 trajnetv
<span class="nb">source </span>trajnetv/bin/activate
<span class="c">## Download Requirements</span>
<span class="nb">cd </span>trajnetplusplusbaselines/
pip <span class="nb">install</span> <span class="nt">-e</span> <span class="nb">.</span>
<span class="nb">cd</span> ../trajnetplusplusdataset/
pip <span class="nb">install</span> <span class="nt">-e</span> <span class="nb">.</span>
pip <span class="nb">install</span> <span class="nt">-e</span> <span class="s1">'.[test, plot]'</span>
</code></pre></div></div>
<p>Alright, our repositories are now setup!</p>
<h2 id="dataset-preparation">Dataset Preparation</h2>
<p><a href="https://github.com/vita-epfl/trajnetplusplusdataset">Trajnetplusplusdataset</a> helps in creating the dataset splits to train and test our prediction models. In this example, we will be using the ORCA simulator for generating our synthetic data.
Therefore, we will setup the simulator with the help of this <a href="https://github.com/sybrenstuvel/Python-RVO2">wonderful repo</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Download Repository</span>
wget https://github.com/sybrenstuvel/Python-RVO2/archive/master.zip
unzip master.zip
<span class="nb">rm </span>master.zip
<span class="c">## Setting up ORCA (steps provided in the Python-RVO2 repo)</span>
<span class="nb">cd </span>Python-RVO2-master/
pip <span class="nb">install </span>cmake
pip <span class="nb">install </span>cython
python setup.py build
python setup.py <span class="nb">install
cd</span> ../
</code></pre></div></div>
<p>We also download the Social Force simulator available at <a href="https://github.com/svenkreiss/socialforce">this repository</a>.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c">## Download Repository</span>
wget https://github.com/svenkreiss/socialforce/archive/refs/heads/main.zip
unzip main.zip
<span class="nb">rm </span>main.zip
<span class="c">## Setting up Social Force</span>
<span class="nb">cd </span>socialforce-main/
pip <span class="nb">install</span> <span class="nt">-e</span> <span class="nb">.</span>
<span class="nb">cd</span> ../
</code></pre></div></div>
<p>Now, we will generate controlled data using the ORCA simulator. We will generate 1000 scenarios of 5 pedestrains moving in an interactive setting.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## Destination to store generated trajectories
</span><span class="n">mkdir</span> <span class="o">-</span><span class="n">p</span> <span class="n">data</span><span class="o">/</span><span class="n">raw</span><span class="o">/</span><span class="n">controlled</span>
<span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetdataset</span><span class="p">.</span><span class="n">controlled_data</span> <span class="o">--</span><span class="n">simulator</span> <span class="s">'orca'</span> <span class="o">--</span><span class="n">num_ped</span> <span class="mi">5</span> <span class="o">--</span><span class="n">num_scenes</span> <span class="mi">1000</span>
<span class="c1">## To know more options of generating controlled data
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetdataset</span><span class="p">.</span><span class="n">controlled_data</span> <span class="o">--</span><span class="n">help</span>
</code></pre></div></div>
<p>By default, the generated trajectories will be stored in <em>‘orca_circle_crossing_5ped_1000scenes_.txt’</em>. Procedure for extracting publicly available datasets can be found <a href="https://github.com/vita-epfl/trajnetplusplusdataset/blob/master/README.rst">here</a>. Also, the goals of the generated trajectories are stored in the <em>‘goal_files’</em> folder under the same name as the .txt file.</p>
<p>We will now convert the generated ‘.txt’ file into the TrajNet++ data structure format. Moreover, we will choose to select only interacting scenes (Type III) from our generated trajectories. More details regarding our data format and trajectory categorization can be found on our <a href="https://www.aicrowd.com/challenges/trajnet-a-trajectory-forecasting-challenge">challenge overview page</a>.</p>
<p>For conversion, open the <em>trajnetdataset/convert.py</em>, comment the real dataset conversion part in main() and uncomment the below given snippet.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## Run the conversion
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetdataset</span><span class="p">.</span><span class="n">convert</span> <span class="o">--</span><span class="n">linear_threshold</span> <span class="mf">0.3</span> <span class="o">--</span><span class="n">acceptance</span> <span class="mi">0</span> <span class="mi">0</span> <span class="mf">1.0</span> <span class="mi">0</span> <span class="o">--</span><span class="n">synthetic</span>
<span class="c1">## To know more options of converting data
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetdataset</span><span class="p">.</span><span class="n">convert</span> <span class="o">--</span><span class="n">help</span>
</code></pre></div></div>
<p>Once the conversion process completes, your converted datasets will be available in the <em>output</em> folder. Trajnetplusplustools provides the following utilities to understand your dataset better. To visualize trajectories in terminal in MacOS, I use <a href="https://github.com/daleroberts/itermplot">itermplot</a>.</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## obtain new dataset statistics
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetplusplustools</span><span class="p">.</span><span class="n">dataset_stats</span> <span class="n">output</span><span class="o">/</span><span class="n">train</span><span class="o">/*</span><span class="p">.</span><span class="n">ndjson</span>
<span class="c1">## visualize sample scenes
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetplusplustools</span><span class="p">.</span><span class="n">trajectories</span> <span class="n">output</span><span class="o">/</span><span class="n">train</span><span class="o">/*</span><span class="p">.</span><span class="n">ndjson</span> <span class="o">--</span><span class="n">random</span>
<span class="c1">## visualize interactions (Default: Collision Avoidance)
</span><span class="n">mkdir</span> <span class="n">interactions</span>
<span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetplusplustools</span><span class="p">.</span><span class="n">visualize_type</span> <span class="n">output</span><span class="o">/</span><span class="n">train</span><span class="o">/*</span><span class="p">.</span><span class="n">ndjson</span>
</code></pre></div></div>
<p>Finally, move the converted data and goal files (if necessary) to the trajnetbaselines folder.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="nb">mv </span>output ../trajnetplusplusbaselines/DATA_BLOCK/synth_data
<span class="nb">mv </span>goal_files/ ../trajnetplusplusbaselines/
<span class="nb">cd</span> ../trajnetplusplusbaselines/
</code></pre></div></div>
<p>Now that the dataset is ready, its time to train the model! :)</p>
<h2 id="training-models">Training Models</h2>
<p>Training models is more easier than generating datasets in Trajnet++ !
All you got to do is ….</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">trainer</span> <span class="o">--</span><span class="n">path</span> <span class="n">synth_data</span>
</code></pre></div></div>
<p>…. and your LSTM model starts training. Your model will be saved in the <em>synth_data</em> folder within <em>OUTPUT_BLOCK</em>. Currently, models are being saved according to the type of interaction models being used.</p>
<p>In order to train using interaction modules (eg. nearest-neighour encoding) utilizing goal information, run</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">trainer</span> <span class="o">--</span><span class="n">path</span> <span class="n">synth_data</span> <span class="o">--</span><span class="nb">type</span> <span class="s">'nn'</span> <span class="o">--</span><span class="n">goals</span>
</code></pre></div></div>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1">## To know more options about trainer
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">trainer</span> <span class="o">--</span><span class="n">help</span>
</code></pre></div></div>
<h2 id="evaluating-models">Evaluating Models</h2>
<p>One strength of TrajNet++ is its extensive evaluation system. You can read more about it in the <a href="https://www.aicrowd.com/challenges/trajnet-a-trajectory-forecasting-challenge">metrics section here</a>.</p>
<p>To perform extensive evaluation of your trained model. The results are saved in Results.png</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">trajnet_evaluator</span> <span class="o">--</span><span class="n">path</span> <span class="n">synth_data</span> <span class="o">--</span><span class="n">output</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">synth_data</span><span class="o">/</span><span class="n">lstm_vanilla_None</span><span class="p">.</span><span class="n">pkl</span>
<span class="c1">## To know more options about evaluator
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">trajnet_evaluator</span> <span class="o">--</span><span class="n">help</span>
</code></pre></div></div>
<p>To know more about how the evaluation procedure works, please refer to this <a href="https://github.com/vita-epfl/trajnetplusplusbaselines/blob/master/evaluator/README.rst">README</a>.</p>
<h2 id="visualize-models">Visualize Models</h2>
<p>Visualize learning curves of two different models</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">trajnetbaselines</span><span class="p">.</span><span class="n">lstm</span><span class="p">.</span><span class="n">plot_log</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">synth_data</span><span class="o">/</span><span class="n">lstm_vanilla_None</span><span class="p">.</span><span class="n">pkl</span><span class="p">.</span><span class="n">log</span> <span class="n">OUTPUT_BLOCK</span><span class="o">/</span><span class="n">synth_data</span><span class="o">/</span><span class="n">lstm_goals_nn_None</span><span class="p">.</span><span class="n">pkl</span><span class="p">.</span><span class="n">log</span>
</code></pre></div></div>
<p>Visualize predictions of models</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># python -m evaluator.visualize_predictions <ground_truth_file> <prediction_file>
</span><span class="n">python</span> <span class="o">-</span><span class="n">m</span> <span class="n">evaluator</span><span class="p">.</span><span class="n">visualize_predictions</span> <span class="n">DATA_BLOCK</span><span class="o">/</span><span class="n">synth_data</span><span class="o">/</span><span class="n">test_private</span><span class="o">/</span><span class="n">orca_five_synth</span><span class="p">.</span><span class="n">ndjson</span> <span class="n">DATA_BLOCK</span><span class="o">/</span><span class="n">synth_data</span><span class="o">/</span><span class="n">test_pred</span><span class="o">/</span><span class="n">lstm_vanilla_None_modes1</span><span class="o">/</span><span class="n">orca_five_synth</span><span class="p">.</span><span class="n">ndjson</span> <span class="o">--</span><span class="n">n</span> <span class="mi">10</span> <span class="o">--</span><span class="n">random</span>
</code></pre></div></div>
<h1 id="done-done">Done Done</h1>
<p>I hope this blog provides you with the necessary kickstart for using TrajNet++. If you have any questions, feel free to post issues on <a href="https://github.com/vita-epfl/trajnetplusplusbaselines">Github</a>. If you liked using TrajNet++, a token of appreciation to parth.kothari@epfl.ch would really go a long way for me ! :)</p>Parth Kothariparthkothari811@gmail.comIn this blog post, I provide a kickstarter guide to our recently released TrajNet++ framework for human trajectory forecasting. We recently released TrajNet++ Challenge for agent-agent based trajectory forecasting. Details regarding the challenge can be found here. This post will focus on utilizing the TrajNet++ framework for easily creating datasets and learning human motion forecasting models.