- Gemini Observatory Archive - Help

Gemini Observatory Archive - Help

Welcome to the Gemini Observatory Archive. We have tried to make the system as simple to use as possible.

Searching

There is only one search form for the archive. The archive can be found at https://archive.gemini.edu. Enter appropriate text into the relevant boxes, and/or make selections in the relevant pull downs, and hit the search button. Leave any fields you do not want to search on blank or set to Any. For more information on each field, mouse over the help text which appears in parenthesis after the search field.

You can bookmark, save, or share your search if you like.

Searching for data for a particular program

If you are the PI or a Co-I of a program, or otherwise wish to search for data taken for a particular program ID, enter the full program ID in the Program ID search field and hit search. You don't need to enter anything else. Assuming you have a reasonable internet connection, results should appear within a few seconds. Note that this is an exact match search, you have to enter the full program ID, and wildcards will not work in here.

Searching by PI/CoI name and Science Program Title

You can also search by PI / Co-I name and words from the science program title. Enter these in the appropriate boxes on the search form. If you just click the Search button, it will directly search for data associated with science programs that match the name and/or title searches. This may be what you want, but probably a more useful way to use this feature is to click the "Search Programs Only" button - this will bring up a list of matching science programs, from which you can click the program IDs to search for data from that program. See also the section in this document about Searching for Science Programs.

Searching by instrument and instrument configuration

To search for data from a particular instrument, select that instrument in the instrument pull-down. If you select one of our facility instruments, an extra part of the form will appear which allows you to search for data taken in specific configurations of that instrument, for example with a certain filter, diffraction grating or readout mode. The options are specific to each instrument and cover the major modes of that instrument.

Searching by object name or sky-coordinates

If you simply enter some text in the Target Name box, the archive will search for observations that used that specific target name. You can use '*' as a wildcard - see the help text on the search field for details, but note that you are searching based on what was entered as the target name in the Phase-II. Obviously, many astronomical objects have multiple names, and also there's nothing to stop the PI calling their target whatever they like. If you are looking for data on a particular object on the sky, you are better off either entering the coordinates directly, or using the name resolver. To use the name resolver, enter your target name in the Target Name field, then select which name resolver you would like to use and click the "Resolve" button. The name resolver will look up your object by name and enter the coordinates in the RA and Dec fields for you. Your search will then be based on those sky coordinates, not the literal name of the target.

Proprietary Coordinates

The default for all gemini data is that all metadata is immediately public, only the actual pixel values are considered proprietary. A few campaign science projects are granted a special status where the coordinates of their observations are also considered proprietary. This is typically where multiple observations of the same target are necessary to confirm a result, and the concern is that a third party who notices which targets are being heavily re-observed could scoop the result. Once the proprietary period has expired, both the coordinates and pixel values of these data become public. While they are still proprietary, they will never show up in searches by any sky coordinate (RA, Dec, Azimuth, Elevation etc etc). This holds true even if you are logged in as a user that has proprietary access to that data. In that case, full details of the data will show up if you find the data by other means (eg by program ID or observation date), but the data will still not show up if you search by coordinates.

More generic searches

You can of course search by any combination of items on the search form, including any combination of the above. If you wish, you can subsequently narrow down your search further - for example you could enter a date range if you were only interested in data taken between certain dates, or if you are only interested in science data (as opposed to calibration and acquisition data), you can select science in the Obs Class pull-down. Simply fill out whichever combination of fields you would like to search by and click search.

Results Table features

In the search results table, you can click on the filename of a data file to see the full FITS header of that file in a new browser tab. Also if you have access to that data file you will see a [P] in the download column - this shows you a preview of that data file in your browser window. You can click on the preview image to make it go away, or while it is still up you can click on the [P] link for a different file to see the preview of that file. Some data may not have preview images available, for example certain visitor instrument data or data that was only ingested into the archive a few seconds before you did your search. Most previews just show the raw data, although Mid-IR data taken with chopping will show you the sum and the difference of the chop beams to aid visualization of the data.

Proprietary data to which you do not have access will not show the Preview and Download ([P] and [D]) links but instead will show the date that the data becomes public. You can mouse over this data for a tool-tip giving an explanation of the proprietary status.

In the data label column for some data sets you may see [A] links. These display the Program Information and Abstract for that science program. This is the same information accesible via the Science Program Search feature, but in a pop-up form similar to the preview images. Some data sets will also show an [L] link, which displays observing log comments that were made by the observer or data analyst regarding data sets in that observation.

The rest of the columns in the table that are hyperlinks will add the thing you clicked on as a search to your search form. So for example if your search returns data from a number of different instruments, and you click on the instrument name in one of the results, you will refine your search to that instrument only. Likewise, you can click an observation class or observation type to narrow your search down to only that observation class or type.

The links in the Data Label are a little more subtle. Gemini Data Labels are composed of the Program ID followed by the Observation number and finally the data set number. These are three separate links in the data label. Clicking on any of these will add that as a search constraint. So for example if the data label is GN-2010A-Q-11-22-33 then clicking GN-2010A-Q-11 will add that program ID as a search constraint, clicking the 22 will add Observation ID GN-2010A-Q-11-22 as a search constraint, and clicking the 33 would add the complete data label as a search constraint, which should only find files with that exact data label.

Note that clicking the links in the results table disregards any changes you have made to the fields in the form since your last search - if you edit the fields in the form, you have to click the "Search" button for that to take effect.

To undo any search selections made by clicking the links, clear the fields in the search form and click the Search button. Note that observation ID and data label share the program ID form field, so for example to remove an observation id selection but keep the program id selection, just delete the last part to make it back into the program ID.

The number of rows in the results table will be truncated if your search returns too many results. The limit depends on whether your search is constrained or not - searches that include a date or date range, or program ID for example by definition have a limited number of possible results and in those cases you will see up to 10,000 results. However a search that does not have such constraints will be cut off at 500 results.

You can also sort the results table by clicking the small up and down arrows in the column headings. Note that if your results were truncated, you may well see different results by sorting, as the sort applies to the search before it is truncated.

Successive Searching

When you fill out the search form and click the Search button, you'll see your search results below the filled out search form. If you look at the results and realize you would like to narrow your search down further, you can add extra criteria to the search by filling in additional fields in the form and clicking Search again. Conversely, if you look at your results are realize that your search is over-constrained, you can broaden it by deleting the text in any of the text fields, or by setting one or more of the pull-downs back to "Any".

If you start changing things on the form and realize that you've made a mistake, you can click the "Reset to previous search" button to reset the form fields to how they were last time you clicked search. There's also a "Start Over" button that will completely reset the form to the blank defaults.

Bookmarking, Saving, and Sharing Searches

When you do a search, the search terms are encoded in the URL of the search results. This means that you can Bookmark the results page in your browser and revisiting that bookmark will essentially re-run the same search on the archive and show you up to date results. So for example if you are PI on a program, you could bookmark the search for data for your program and when you visit that page you'll see an up to date listing of your data. The bookmarked page is just the same as if you'd filled out the search form again - all the features such as successive searching still work.

You can also copy the URL of a search from your browser and store it (eg in a text file or document) or share it (eg by email or social media) for example with collaborators or colleagues. It's just a URL like any other web page, that when visited behaves just like you had re-entered the original search terms in the form and clicked Search.

Searching for Calibrations

You can search for calibration data directly by setting the Observation Type and/or Observation Class fields to match the calibrations you want to search for - for GMOS BIASes are taken with observation type "BIAS" and observation class "dayCal". If you are looking for processed calibrations, then you can select that in the Raw / Reduced pulldown. You probably also want to select the instrument and set some of the instrument configuration fields to match what you're looking for (eg the filter, grating, or exposure time).

The archive will also automatically associate calibrations for you when you search for data. The search results area under the form is has three tabs - if you click on the "Load Associated Calibrations" the archive will go through your search results and find the best match calibration files for all relevant baseline calibration types for each file. This can take a few seconds, but once it's loaded you can flip back and forth between the tabs quickly.

Data Access and Proprietary Period

Data files have a proprietary period during which they are only available to the PI who proposed the observations, or their delegates. The proprietary period varies, but for regular Queue and Classical data it is currently 12 months, having been decreased from the previous default of 18 months by the Gemini Board during the renewal of the international agreement with effect from semester 2016A. Some programs such as Science Verification programs may have shorter proprietary period, and Calibration data generally have no proprietary period.

Non-proprietary data is world public. You do not need to be logged in to the archive to download it.

To access proprietary data in the archive, you must be logged in to the archive and you must have registered your program ID with your archive account. Details on that follow.

User Accounts and Registering Programs

If you are not logged in to the archive, the top right of the search form web page will show a user icon and the message "Not logged in". If you mouse over this you will see a drop down menu with options to request an account or to login. To request an account you will need to provide a valid email address. The system will email you a confirmation link that you must click within 15 minutes. You will be then led through setting a password and logging in. Please note that archive accounts are for individual use only. You should have your own personal account and should not share the password with anyone. Multiple accounts can register for proprietary access to the same program, there is no need to share user accounts.

If you are logged into the archive, the top right of the search form web page will show your username. If you mouse over it you will get a drop down menu with options to log out, change your password, or update your programs.

Note that once you log in to the archive from a certain computer or browser, you will stay logged in from that browser for up to a year or until you log out. Leaving the site and coming back should not require you to log back in. Obviously, you should be aware of this when using public or shared computers. Note that if you log out from one browser, all your archive sessions will be logged out. So if you think you might have left some system logged in that you didn't mean to, you can just log out from somewhere else and it will log out all your sessions.

Registering programs for Proprietary Data Access

Once you are logged in, you can register a Gemini Program ID with your archive account to get access to proprietary data from that program. To do so, select the "My programs" link in the drop down under your username. Inside that page, you enter the Gemini Program ID and provide the Program Key - this is the key that was provided in your time award notification email. PIs should provide the program key to Co-Is, collaborators, students, or anyone who they wish to have access to the proprietary data.

Once you have registered a program with your account, you will be able to preview and download proprietary data from that program.

Downloading Data

If your search constrained the number of results (see earlier) then right at the end of the results page there will be a "Download all" button. Alternatively, you can go through the table and click the button in the download column for the files you want and click the "Download Marked Files" button. If you want a lot of files but not all of them, you can probably constrain your search to just the ones you want and then use the Download All feature. Either way, this will send you a tar file containing the bzip2 compressed tar files, a README.txt file, and an md5sums.txt file. bzip2 is a standard compression tool installed on almost all Linux and Mac computers by default and available for almost all platforms. The uncompress command is usually bunzip2. The README file gives you a summary of what was downloaded, and the md5sums.txt file allows you to verify that your files were not corrupted during the transfer - see instructions in the readme file.

The download system should activate your browser's regular Save-As feature, no additional plugin or app is required. The system will suggest a default filename for the download based on the search criteria, your browser may allow you to modify that before the file is saved. Some browsers will prompt for the directory where you want to save the file, some will default to a Downloads directory or similar.

Searching for Observation Logs

You can also search for observation logs both directly and by association. To search directly, fill out the program ID and/or date search fields and click the "Search Obslogs Only" button. To see observation logs associated with a regular search for data, click the "Load Associated Observation Logs" tab in the results area, and the archive will show you all the Observation Logs associated with data in your search results. Once you have a listing of Observation Logs, you can click on a filename to view it in a new tab in your browser, or you can use your browser's "Save As" function to save it to local disk.

Searching for Science Programs

To search for science programs directly, fill out some combination of PI/CoI Name, Program Title, or Program ID fields and click the "Search Programs Only" button. This will generate a table of matching science programs, with the program ID link taking you to a search for data from that program, and the program title link taking you to the information page for that program. The program information page also has information about known publications that are associated with that program, with links to data searches and to ADS.

Miscellaneous file collection

There are some files in the Gemini data collection we do not lend themselves to inclusion in the main archive. Generally this is because they do not have adequate metadata to allow them to be meaningfully searchable using the regular archive interface. These files are grouped togething into tar file packages with a text description which is searchable. They are held on a separate area of the archive site. Files from the visiting DSSI instrument are stored here for example.

Click here to access the miscellaneous file collection.

Accessing the Archive from scripts and the command line

Many URLs to services on the archive can be constructed easily by hand or programmatically, and then accessed using command line tools such as wget or curl, or from a scripting language, for example using the urllib module or similar in python. Further details are provided in the Application Programmer Interface (API) documentation

HELP - Things don't seem to be working

If things really don't seem to be working for you, here's some suggestions:

Browser Compatibility

The site uses CSS widely, and JavaScript to handle some elements of the search form. Older versions of browser software not have support for the features we use. Any common browser from the last few years should be fine. We are aware that browsers from earlier than 2012 may not work well with the arcive web pages.

Can't access proprietary data

In order to access your proprietary data, you must have created an account on the archive, be logged into that account, and you must have registered your Gemini program with your account. Please see the Data Access section above for details.

No results

If you're getting no results when you were expecting to get results, most likely the data you're looking for does not match one of the search criteria you have. Try setting some of the settings in your search form back to "Any" and see if the data show up.

Helpdesk

If you're still struggling, please file a ticket in the Gemini Helpdesk under the "Gemini Observatory Archive" section.