How do I Import Existing Data into a REDCap Project?
Here is a summary of REDCap Importing options. There are two primary options.
1. Manual Import into REDCap via a CSV (or excel file saved as a CSV).
2. API based import (search API in this form for more information about the API tools, or go to the REDCap API Playground)
Notes on Manual Import:
Each project’s import structure is available in REDCap via the menu applications --> Data Import Tool --> Download your Data Import Template.
The data Import file for “Logan: HIRC Working Project List” is attached as an example. The process here is to generate a CSV file in the structure provided by the Data Import tool, which is essentially just a file with the variable names in the top row. You would then upload this file into REDCap periodically via Data Import Tool --> Choose File --> Upload File --> Review the changes the file is set to make ? Approve the upload of the file.
Notes on API:
REDCap will support a variety of languages and has a point and click interface to create a project and user-specific API Keys. You can review this tool via Applications ? API or Application ? API playground to see standard API methods associated with JSON, CSV, or XML formatted data in the languages PHP, PERL, Python, Ruby, Java, R, or cURL.
Comparing and Contrasting the two methods:
Although the API is often considered the more desirable choice, as a periodic manual upload sounds cumbersome, the periodic manual upload has some advantages over the API. Most significantly, the manual upload offers an additional data adjudication step that the API does not have. The data adjudication step in the manual import allows users to review the new data going into the study, it allows a human to logic check which data is over-written, and it allows a human to logic check unchanged and new data on a per record basis. This is done by a color-coded table which lays out all of the overwritten, new and unchanged data in an easy to view format. Perfect for a quick logic check before going into your database.
The API, on the other hand, acts automatically, it does not allow for data adjudication by a human before the data is imported. However, if you completely trust the source of data and just want REDCap to import in from the “Source of Truth” the API could work. In most cases, data setups using the API, consider REDCap the “source of truth” and run data checks in SAS or R on the data to adjudicate via human logic check before importing into REDCap via API. This step is important to most studies and databases because there is always the potential for a small error in a data table (record ID row shift, etc.) to completely change the data locations and write over existing data in REDCap.
Manual Import guide below:
This guide is useful for the following...
a) Monthly Imports into a centralized REDCap project from a different data source
b) Manual imports from one REDCap project to another
c) Preparing manual export of select data from your REDCap database to give to another institution to fix
If you need to view your own project’s REDCap Data Import Template, Under Applications Menu → Select Data Import Tool --> Download your Data Import Template.
The Data Import Template provided by your centralized database or that is associated with your project will look like this. This is an excel sheet with the variable names across the top.
Ultimately data will be imported into the centralized REDCap database by matching these data location columns (variables), placing your data in the correct location and sending
this database to a member of the centralized database’s team, or upload it into your own project for import.
If you are doing a monthly upload to a centralized database then it will be most efficient to utilize reports to make pulling data on a periodic basis easier and done with one set-up.
Our goal will be to make a report that has as all the variables, in the same order, as the data import template.
If you are importing a specific instrument(s) you can bring that instrument into the report all at once.
Our goal is the get as close as we can to matching the variables and the order of the variables in the data import template.
Depending on your data collection workflow, it might be important to add additional filters to the data.
Some examples of additional filter options that might be useful include:
1)Filtering by Date Range (if you are providing a centralized database with a monthly report, you can adjust this each month to select only the monthly data)
2)Ordering the data by a specified variable, for example, record ID, date, name, etc.
Users can view the report. Compare the report against the Data Import Template. The goal is the get the data location matching as closely as possible. The REDCap Data Import Template does not require that every field be filled out in order to complete the upload. Just match as closely as possible.
The centralized database staff will need to review the data and ensure that the data, have the required fields, record ID naming conventions, and required fields for upload into the centralized database.
When you’re satisfied with your report, select the export report.
We recommend selecting Microsoft Excel labels or raw day for easiest Report and Data Import Tool Matching.
Download the excel file
Copy the data from your report into the data import template.
Logic check, once more, the data location in the data import template. Does each variable line up? Is each column of data in the correct location?
Once you have the data lined up, you can send it to your centralized database or you can upload it into your REDCap Project. Just save your file. Go to the Data Import Tool. Select Choose File.
Select your saved data Import template, and select upload file. REDCap will review the data, give you a summary of what data is being overwritten, what data is new, and what data is already in the database. It will also summarize allowable errors and/or errors that prevent the upload from happening. You can decide to move forward or to not import the files.