ROUGH DRAFT authorea.com/3122
Main Data History
Export
Show Index Toggle 0 comments
  •  Quick Edit
  • Physics 90r Update 1

    Research Plan

    Original

    1. Create and clean one master dataset
    2. Use SQL to manage the dataset and Pandas for small-scale exploratory analysis
    3. Carefully keep track of filters applied to the master dataset in exploratory analysis
    4. Visualize data in many two-dimensional spaces
    5. Identify and model clusters in multidimensional space
    6. Build empirical SED's

    Updated Plan

    1. The creation of a clean database was only the first step in the objectives list.
      • WISE table built
      • 2MASS have the data => build database+schema.
      • specify primary and secondary keys to link WISE and 2MASS.
    2. concrete evidence that you know how to use the database you created.
      • couple simple SQL queries showing that you have keys figured out.
      • a query that uses nulls.
    3. exporting data to Pandas for visualization
    4. creating 2dimensional plots that demonstrate that you have a working knowledge of the two linked databases.

      • magnitude versus error color coded by flag.
      • spatial plot RA versus DEC. for different magnitude or color cuts.
      • color code the symbols the spatial plot by a spectrum => color code symbols by size "color (J-K, K-24)"
      • 2 Color or Color+Magnitude diagrams that use both 2MASS and WISE. K versus K-22 microns uses both datasets.
    5. Glue like stuff.

    Deliverables

    • ipython notebook demonstrating python<=>SQL access and visualization.
      • pip freeze > requirements.txt (share that as well)
      • github repository
    • authorea/ipython notebook documentation.
      • descriptions of scripted access
      • schema creations
      • doc where nulls came from
      • doc where keys came from
    • SQL dump
      • dump SQL and schema.
      • flash drive.

    Learned about postgres, SQL, and RBDMS

    \label{sec:background}

    • PostgreSQL by Douglas and Douglas

    • Practical PostgreSQL by Drake and Worsley

    Installed postgres

    \label{sec:installation}

    • Downloaded and installed using graphical installer found at:
      http://www.enterprisedb.com/products/pgdownload.do#osx

    • Told postgres to listen on port 5432

    • Used StackBuilder to install psqlODBC v09.02.0100-1

    • postgres commands live at /opt/local/lib/postgresql93/bin

    • Created a database cluster at /usr/local/pgsql/data

    • Start database server using: /opt/local/lib/postgresql93/bin/postgres -D /usr/local/pgsql/data

    • Created a user, “mattmoellman”, and a test database, “mydb”

    Next steps

    \label{sec:nextsteps}

    Datasets

    \label{sec:datasets}

    APASS

    \label{sec:apass}

    Column Comment psql dtype Null
    RA (decimal degrees)
    RA error (arcsec)
    DEC (decimal degrees)
    DEC error (arcsec)
    nobs (number of observed nights)
    V mag
    Verr mag
    B mag
    Berr mag
    g mag
    gerr mag
    r mag