Physics 90r Update 1

Research Plan


  1. Create and clean one master dataset
  2. Use SQL to manage the dataset and Pandas for small-scale exploratory analysis
  3. Carefully keep track of filters applied to the master dataset in exploratory analysis
  4. Visualize data in many two-dimensional spaces
  5. Identify and model clusters in multidimensional space
  6. Build empirical SED's

Updated Plan

  1. The creation of a clean database was only the first step in the objectives list.

    • WISE table built
    • 2MASS have the data => build database+schema.
    • specify primary and secondary keys to link WISE and 2MASS.
  2. concrete evidence that you know how to use the database you created.

    • couple simple SQL queries showing that you have keys figured out.
    • a query that uses nulls.
  3. exporting data to Pandas for visualization

  4. creating 2dimensional plots that demonstrate that you have a working knowledge of the two linked databases.

    • magnitude versus error color coded by flag.
    • spatial plot RA versus DEC. for different magnitude or color cuts.
    • color code the symbols the spatial plot by a spectrum => color code symbols by size "color (J-K, K-24)"
    • 2 Color or Color+Magnitude diagrams that use both 2MASS and WISE. K versus K-22 microns uses both datasets.
  5. Glue like stuff.


  • ipython notebook demonstrating python<=>SQL access and visualization.
    • pip freeze > requirements.txt (share that as well)
    • github repository
  • authorea/ipython notebook documentation.
    • descriptions of scripted access
    • schema creations
    • doc where nulls came from
    • doc where keys came from
  • SQL dump
    • dump SQL and schema.
    • flash drive.

Learned about postgres, SQL, and RBDMS


  • PostgreSQL by Douglas and Douglas

  • Practical PostgreSQL by Drake and Worsley

Installed postgres


  • Downloaded and installed using graphical installer found at:

  • Told postgres to listen on port 5432

  • Used StackBuilder to install psqlODBC v09.02.0100-1

  • postgres commands live at /opt/local/lib/postgresql93/bin

  • Created a database cluster at /usr/local/pgsql/data

  • Start database server using: /opt/local/lib/postgresql93/bin/postgres -D /usr/local/pgsql/data

  • Created a user, “mattmoellman”, and a test database, “mydb