Philip Morse edited section_Results_subsection_In_the__.tex  almost 9 years ago

Commit id: 95a3936b03ef31732552f685d58465d43b947bb2

deletions | additions      

       

\section{Results}  \subsection{?} \subsection{Traffic Data Compared with MOTA}  In the first phase of optimization, the genetic algorithm was run for the full 10 minutes for each video sequence with a corresponding ground truth. Each generation had 20 specimens, a score threshold of 0.2 for natural deaths, a maximum of 60 percent survivors, a crossover rate of 60 percent and a mutation rate of 5 percent. Each specimen contained a file that was saved and evaluated in terms of traffic count and average speed at the entrance and exit of each lane of the analysis zone, represented in figure (insert figure showing counts/speed).   These results were used to validate the choice of MOTA as a measure of performance. As expected, there is a strong correlation between MOTA and the number of objects tracked due to the nature of MOTA. It is important to note that the total vehicle counts are not necessarily comparable to the number of objects in the ground truth since the genetic algorithm is run on the analysis zones which do not include the whole video. Lower MOTA results also tend to have a larger range in terms of average tracked speed. As the performance increases in lanes that have a significant number of vehicles, the average tracked speeds converge. The two higher resolution cameras (S3V1 and S3V2) had very few results of poor performance. This suggests a lower sensitivity to the chosen parameters used in the genetic algorithm. S1W presents a special case where only 49 percent of specimens had any tracked objects. These two cases are discussed in subsequent sections. 

Performance of the low resolution IP-camera was largely dependent on the feature-quality. In fact, in the case of the winter camera, a MOTA above 0.150 was not found unless the feature-quality was below 0.10 and the best performance came from a feature-quality of approximately 0.01. This suggests that the winter camera should be calibrated separately from any other camera since the most important parameter is directly related to computation time. The range of the feature-quality should also be adjusted for such a calibration. Another point of mention is that the GoPro camera had no strong correlation towards any of the parameters. The implication is that the high resolution allows for a wide range of parameters to have similar performances. The third implication is that certain parameters such as min-feature-distanceklt and min-feature-time could potentially be eliminated from the optimization process due to the lack of correlation on any camera.(Future work: try gopro camera in different weather conditions to see if it is also applicable, try to calibrate to reduce computation time since the downside is that it takes very long to find trajectories.)  \subsection{?} \subsection{Over-fitting Analysis}  The second phase of the validation process was to evaluate the performance of optimized parameters compared to the default parameters. The genetic algorithm was run for the first five minutes of each video file for between 10 and 30 generations over the course of one to three days depending on the site. Table (insert table about optimized parameters) is a summary of the optimal parameters found for these specific sequences. Some parameters such as 'min-feature-distanceklt', 'mm-connection-distance' and 'mm-tracking-error' seem to have converged towards similar values on three of the cameras. However, it is apparent that in most cases the optimal solution is unique.  

This was done in an attempt to verify whether or not overfitting was occurring. It was found that, while the improvement was sometimes greater for the sequence on which the optimization was executed, there was a significant positive upward trend in performance on the control set. A point of mention is that the MOTA of the close and far zones do not always average to the same result as the full analysis zone. This is explained by the small errors created at the borders of the analysis zone as vehicles enter and leave.   \subsection{Optimized Parameters Applied on Seperate Datasets}  The third phase was to take the parameters from the second phase and apply them on the other annotated videos. Table (insert table) is an evaluation of the MOTA on each video sequence for each set of optimized parameters.  The MOTA was compared both as an absolute value as well as a percentage of the known highest performance value to avoid potential bias from the site selection or selection,  the number of vehicles annotated, and the vehicle composition,  as discussed previously. Trajectories generated in winter conditions were almost non-existent in most cases, leading to MOTA values far outside the acceptable range. Inversely, using the set of parameters obtained from the winter camera resulted in a performance worse than the default tracking in certain cases. This suggests the need for a separate set of parameters for different weather conditions. However, in the case of good weather conditions, a result of at least around  90 percent of optimal can be expected from a set of optimized parameters. Combining the results from this section with the traffic counts in the first section, it can be inferred that there is a need for calibration in certain circumstances where simply using the default parameters would lead to an estimation of a flow rate which is significantly lower than reality.  \subsection{Sources of Error}