Shaswat Patel

Investigating the safety risks of episodic memory in AI agents.

1. Prof. He He: Do coding agents lead to long term maintainability issues of the underlying code? Yes, so how do we measure it? How do we build a work-able environment that can be used to test the underlying observation? 2. Prof. Eunsol Choi: Retrieval heads are quite important for tasks such as long-form QA and are also crucial for all the RAG-based systems. Furthermore, retrieval heads contribute significantly towards performance gains due to CoT. But how do they emerge? How do they perform across multiple languages? Are they language agnositic?

Event Recommendation GPT: A search and discovery utility tool for real-life events. The system consists of: 1. RAG: Implemented a dense retrieval pipeline leveraging OpenAI embeddings, followed by re-ranking via Jina AI’s cross-encoder, incorporating multi-dimensional signals such as venue information and out-of-distribution scoring to prioritize unique and diverse events. Implemented meta-data extraction and filtering. 2. Memory Module: Implemented a custom memory architecture to store long-term user preferences (e.g., preferred genres, locations). The implementation includes tool-calling for CRUD operations on memory. The current module is 85% accuracy at extracting user preferences. 3. Multi-Turn Summarizer: Implemented standard techniques to support multi-turn conversation by maintaining state of the conversation via summarizer. 4. ReAct Tool-Calling Framework: To support multiple queries, the RAG pipeline is encapsulated via a ReACT tool-calling framework. Queries include: weekly calendar creation and daily planner which includes restaurants, activities and events. 5. Test dataset: Automatic pipeline to curate and store events for testing different components of RAG pipeline. The pipeline also includes E2E testing via GEval and DeepEvals.

My team and I leveraged a combination of Natural Language Processing techniques (BioBERT, GPT-3.5, Mistral, etc.) and Machine Learning methods (Logistic Regression, SVM, Gradient Boosting, etc.) to tackle biomedical tasks such as predicting sepsis, abnormal shock index, mortality, and length of stay. We successfully published our work on Abnormal Shock Index Prediction (ShockModes), and our other projects include analyzing bias in large language models (LLMs).

Focused on the field of Visual, Audio, and Audio-Visual Speech Recognition Systems. My work involved extending the Wav2vec-U approach by incorporating unsupervised visual speech recognition capabilities, which we termed Video2vec-U. The modifications included integrating a ViT model for extracting visual features, segmenting these into viseme units, and generating viseme sequences using a specialized generator.

1. Machine Learning Implemented a lightweight ML solution for clinical outcome prediction. We used textual EHR data instead of vitals, structured the data by extracting key information(diseases and drugs). Preprocessed the feature list using regex and clustered similar textual embeddings for drugs and diseases to reduce feature set. Trained ML models like LR, SVM, GBBoost, AdaBoost, etc. achieved comparable results to BERT and BioBERT based classifiers. Used multi-gpu setup to extract textual embeddings reducing the time by 40%. Used FIASS for clustering and sklearn for ML Models. 2. Emerging Trends in NLP Working on retrieval heads. 3. Large Language and Vision Models Working on improving multi-image QA ability of LVLM models(LLAVA, QWEN-VL, etc.)