Deyan Ginev edited section_Rebuilding_the_Foundations_The__.tex  almost 9 years ago

Commit id: 2ec5337df90b54bd27a468173e07eeefc346448c

deletions | additions      

       

\item Reusing a general-purpose distributed architecture always required trade-offs that could not fit with the realities of our hardware or administrative restrictions. There was a clear necessity to build our own custom-fit framework, while using as many standard components as possible, to minimize overhead and potential sources of bugs.  \item arXiv contains extremely irregular TeX data, guaranteeing that LaTeXML will break in all possible ways it can break. The worker processes distributed CorTeX workers which executed the translation  needed several types of monitoring - timeouts, memory limits, and even external cleanup of runaway/zombie children (usually related to graphics conversion via ImageMagick and GhostScript). \item Even when distributing inside our own uni's compute cluster (I was gracefully provided 600 CPUs from \href{http://www.techrepublic.com/resource-library/casestudies/dell-adds-computing-muscle-to-hulk-high-performance-cluster-at-german-university-in-five-working-days/}{HULK} on an on-demand basis), there will be occasional network failures, especially related to server load.   

Let's get down to it.