Video-based transportation data collection

Methods for the detection, tracking and classification of road users

• more recent deep learning-based methods
  • https://github.com/timmeinhardt/trackformer https://github.com/ifzhang/ByteTrack
  • test better optical flow methods (http://sintel.is.tue.mpg.de/results): https://learnopencv.com/optical-flow-using-deep-learning-raft/#raft
• Feature-based tracking is the main and a relatively easy detection and tracking algorithm (ideas for improvements)
  • my paper http://n.saunier.free.fr/saunier/stock/saunier06feature-based.pdf and the paper it is based upon http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.49.599&rep=rep1&type=pdf
  • Some slightly older information http://wiki.polymtl.ca/transport/index.php/FeatureBasedTracking
  • Other features: Predator (video, code source)
  • Feature performance comparison http://vision.middlebury.edu/flow/ (with links to implementations)
• Background subtraction: the most common method is based on a mixture of Gaussians and available in OpenCV http://www.ai.mit.edu/projects/vsam/Publications/stauffer_cvpr98_track.pdf; lit review in http://staff.it.uts.edu.au/~massimo/BackgroundSubtractionReview-Piccardi.pdf
  • Emvisi2: A background subtraction algorithm, robust to sudden light changes http://cvlab.epfl.ch/software/emvisi2/index.php
  • kernel density estimation based background subtraction http://cvlab.epfl.ch/software/emvisi2/index.php
• Object detection/classification: http://lear.inrialpes.fr/people/triggs/pubs/Dalal-cvpr05.pdf, available in OpenCV, which can be used for tracking by detection, e.g. in http://www.vision.ee.ethz.ch/showroom/tracking/
  • tracking by detection: http://www.vision.ee.ethz.ch/showroom/tracking/
  • Pose Estimation for Category Specific Multiview Object (Cars) Localization http://cvlab.epfl.ch/publications/publications/2009/OzuysalLF09.pdf
  • Efficient 3D Object (Cars) Detection using Multiple Pose-Specific Classifiers http://www.bmva.org/bmvc/2011/proceedings/paper20/paper20.pdf

• Reconnecting trajectories in cases of lost trajectories within the same camera view or across cameras
• Volume estimation: the basic method relies on background subtraction and fitting volumes for the various road users (rectangular cuboid/prism for vehicles and cylinder for pedestrians). Some initial papers are http://www.sciencedirect.com/science/article/pii/S1077314207000392, http://www.springerlink.com/content/tr6558j731331m78/, http://iris.usc.edu/outlines/papers/2005/song-nevatia-iccv.pdf
• Video annotation and semi-automated tracking for performance evaluation

• Tracking vehicles from the air http://www.edwardrosten.com/work/vehicle.html

Other resources

• TRB SHRP2 report on Site-Based Video System Design and Development http://onlinepubs.trb.org/onlinepubs/shrp2/SHRP2_S2-S09-RW-1.pdf
• Zu Kim at California PATH http://gateway.path.berkeley.edu/~zuwhan/
• EPFL, in particular Pascal Fua, http://cvlab.epfl.ch/software/index.php
  • Multiple Instance Learning http://vision.ucsd.edu/~bbabenko/project_miltrack.shtml
  • POM: Occupancy map estimation for people detection http://cvlab.epfl.ch/software/pom/index.php
• ETH work http://www.vision.ee.ethz.ch/
  • Online boosting trackers http://www.vision.ee.ethz.ch/boostingTrackers/
  • Work by Leibe and Van Gool http://www.vision.ee.ethz.ch/members/get_member.cgi?id=1 http://www.vision.ee.ethz.ch/publications/pub_readall.cgi?lang=en&year1=&year2=&authors=leibe&keywords=
  • Linear Trajectory Avoidance - A Pedestrian Motion Model http://people.ee.ethz.ch/~pestefan/lta/
• Mohan Trivedi (Computer Vision and Robotics Research Laboratory http://cvrr.ucsd.edu/) and his former student Brendan Morris http://www.ee.unlv.edu/~b1morris/
• Minnesota lab for ITS and video analysis http://airvl.cs.umn.edu/ (work by Masoud, Papanikonikolopoulos et al.)
• Gerard Medioni http://iris.usc.edu/people/medioni/index.html http://iris.usc.edu/USC-Computer-Vision.html
• Greg Mori http://www.cs.sfu.ca/research/groups/VML/MMTrack.html (see references on online tracking
• Ram Nevatia http://iris.usc.edu/Projects/detect/detection.html http://iris.usc.edu/Outlines/Paper-track.html
• Rabaud and Belongie, Counting Crowded Moving Objects http://vision.ucsd.edu/~vrabaud/
• Literature reviews: A Review of Computer Vision Techniques for the Analysis of Urban Traffic http://dx.doi.org/10.1109/TITS.2011.2119372, Object Tracking: A Survey, http://dx.doi.org/10.1145/1177352.1177355
• Computer Vision Algorithm Implementations http://www.cvpapers.com/rr.html (see in particular object detection and tracking)
• Minnesota: Practical Methods for Analyzing Pedestrian and Bicycle Use of a Transportation Facility www.lrrb.org/pdf/201006.pdf
• My other delicious links: http://delicious.com/saunier/cv

Software Development

Because I am relying on OpenCV for computer vision functionality, and because C++ is fast, the previous software was written in C++. For the same reasons, the core most computationally intensive functions should be written in C++ (although the python wrappers are more and more usable). The most up to date documentation is at http://opencv.itseez.com.

The platform of choice for development is Linux (e.g. the Ubuntu distribution). I would like to have the code cross-platform, ie the C++ should compile at least under Windows and Linux, which is not too difficult if using the right tools (g++, make or CMake).

I am If you are not familiar with any of the following topic, please read more:

• C++: see below. I recommend the use of smart pointers for ease of memory management (avoiding memory leaks in a program meant to process hours of video without crashing is essential). See Boost shared pointers http://www.boost.org/doc/libs/1_46_1/libs/smart_ptr/smart_ptr.htm
• Version Control: I refuse to work in teams without software version control. I use mercurial which has a fairly low barrier to entry and good documentation. http://mercurial.selenic.com
• Test-driven development: writing tests takes a bit more time when developing, but helps in testing and ensures that the software still matches its specifications when refactoring later. I have used the Boost test library and it does the job http://www.boost.org/doc/libs/1_47_0/libs/test/doc/html/index.html, Google test is getting famous and is used in OpenCV.
• Computer vision algorithms: see above
• Python is nice for visualization, and the binding to OpenCV seem now robust enough for prototyping.

It is very important for me to develop quality software that can be easily maintained over the long term, hence the emphasis on version control, smart pointers and tests.

Traffic Intelligence

See https://trafficintelligence.confins.net and follow instructions there for installation and use, including OpenCV.

Traffic Intelligence is developed under an open source MIT license and I would like additions to be under the same license.

Cameras

• Vivotek IP8151: used at McGill, issues of framerate at higher resolution
• Panasonic, eg http://www.panasonic.com/business/psna/products-surveillance-monitoring/network-security-cameras/fixed-cameras-color/WV-SP509.aspx
• use portable personal video recorders such as archos (old technology?)
• CMOS sensors, eg http://www.thorlabs.com/NewGroupPage9.cfm?ObjectGroup_ID=2916, http://www.edmundoptics.com/onlinecatalog/Browse.cfm?categoryid=1569

More information on this wiki page

Datasets

Public_Transportation_Datasets

Resources

• Open source computer vision projects
  • OpenCV http://opencv.willowgarage.com/wiki/
  • http://code.google.com/p/opencv-feature-tracker/ (Warning: many bugs, not a good basis to build upon)
  • https://bitbucket.org/Nicolas/trafficintelligence
  • https://bitbucket.org/trajectories/trajectorymanagementandanalysis
• Version Control http://mercurial.selenic.com/
• C++
  • http://en.cppreference.com/w/cpp
  • http://www.parashift.com/c++-faq-lite/
  • C++ code samples (Boost, OpenCV, etc) http://programmingexamples.net/