Main Data History
Show Index Toggle 0 comments
  •  Quick Edit
  • Physics 90r Update 1

    Research Plan


    1. Create and clean one master dataset
    2. Use SQL to manage the dataset and Pandas for small-scale exploratory analysis
    3. Carefully keep track of filters applied to the master dataset in exploratory analysis
    4. Visualize data in many two-dimensional spaces
    5. Identify and model clusters in multidimensional space
    6. Build empirical SED's

    Updated Plan

    1. The creation of a clean database was only the first step in the objectives list.
      • WISE table built
      • 2MASS have the data => build database+schema.
      • specify primary and secondary keys to link WISE and 2MASS.
    2. concrete evidence that you know how to use the database you created.
      • couple simple SQL queries showing that you have keys figured out.
      • a query that uses nulls.
    3. exporting data to Pandas for visualization
    4. creating 2dimensional plots that demonstrate that you have a working knowledge of the two linked databases.

      • magnitude versus error color coded by flag.
      • spatial plot RA versus DEC. for different magnitude or color cuts.
      • color code the symbols the spatial plot by a spectrum => color code symbols by size "color (J-K, K-24)"
      • 2 Color or Color+Magnitude diagrams that use both 2MASS and WISE. K versus K-22 microns uses both datasets.
    5. Glue like stuff.


    • ipython notebook demonstrating python<=>SQL access and visualization.
      • pip freeze > requirements.txt (share that as well)
      • github repository
    • authorea/ipython notebook documentation.
      • descriptions of scripted access
      • schema creations
      • doc where nulls came from
      • doc where keys came from
    • SQL dump
      • dump SQL and schema.
      • flash drive.

    Learned about postgres, SQL, and RBDMS


    • PostgreSQL by Douglas and Douglas

    • Practical PostgreSQL by Drake and Worsley

    Installed postgres


    • Downloaded and installed using graphical installer found at:

    • Told postgres to listen on port 5432

    • Used StackBuilder to install psqlODBC v09.02.0100-1

    • postgres commands live at /opt/local/lib/postgresql93/bin

    • Created a database cluster at /usr/local/pgsql/data

    • Start database server using: /opt/local/lib/postgresql93/bin/postgres -D /usr/local/pgsql/data

    • Created a user, “mattmoellman”, and a test database, “mydb”

    Next steps






    Column Comment psql dtype Null
    RA (decimal degrees)
    RA error (arcsec)
    DEC (decimal degrees)
    DEC error (arcsec)
    nobs (number of observed nights)
    V mag
    Verr mag
    B mag
    Berr mag
    g mag
    gerr mag
    r mag
    rerr mag
    i mag
    ierr mag