Overview
This project focused primarily on pre – processing unstructured data within columns Chief Complaint, Radiology Notes, Discharge Notes, Discharge Instructions respective datasets in order to move onto the normalization and training phase of building a predictive, classification model.
Data & Tools
The Medical Information Mart for Intensive Care dataset was originally sourced from Hugging Face. I have not been able to find the original version I used. Similar versions can be found on the National Institute of Health or Physio.net. It is comprised of six parquet files that can be linked on unique identifiers. The data transformation was conducted in Jupyter Notebook using Python.
Methodology
Libraries pandas, re, numpy, tqdm, torch, transformers, os and ast were used in this project. Each parquet file was loaded using pandas. Radiology Notes and Discharge notes were aggregated for each individual patient encounter. Then columns were selected and combined using a left join. Duplicates were removed and the data was filtered for primary diagnoses only. Finally, tokenization and embeddings retrieval is performed using BERT Base Uncased. The new file with embeddings is then saved to a csv file to scale up in a more appropriate environment.
Code Preview
Full Code Preview
