Lilia Urmazova

- Collaborating in a cross-functional ML team on cutting-edge R&D initiatives, focusing on scalable AI architecture. - Leading LLM evaluation to assess reasoning capabilities, tool-use proficiency, and output quality, selecting and validating evaluation metrics (e.g., G-Eval, RAGas, BLEU/ROUGE, CLIP family, or custom heuristics) - Evaluating AI agent capabilities, including visual, code, and text artifacts, using mutation testing and other reasonable approaches. - Developing end-to-end evaluation pipelines to automate performance tracking and regression testing for machine learning models.

#1 in the CIS for student IT employment. - Leveraged experience in AI testing to contribute to the hiring and onboarding of junior testers, including specialists in AI testing methodologies. - Authored and delivered onboarding educational materials for junior QAs and a textbook covering foundational QA skills (SQL, API, Web, test design, Agile), demonstrating expertise in knowledge transfer and process education.

- Developed and implemented comprehensive testing strategies for AI Vertex search and AI Visual search for merchandisers, including model performance testing, prompt testing, ethics testing, data validation, and bias assessment. - Covered LLM metrics with Python automation and frameworks for AI testing, using LLM-as-a-Judge approach, and model cost analysis. - Prepared and managed large-scale datasets in cloud storage for AI Vertex Search, optimizing data quality and accessibility for model training. - Led a team of 10 QA engineers, ensuring compliance with best testing practices and project deadlines in an Agile environment. - Collaborated with development teams and data specialists to define testing requirements and ensure end-to-end AI functionality quality.

- Led and actively contributed to automation efforts to achieve a best-in-class test pass rate of 100% for Java TestNG API, SQL, and 95% for UI Java Selenium automated tests with BDD (Cucumber), demonstrating a commitment to robust and reliable testing. - Collaborated with the customers’ team to align the comprehensive manual and automation testing strategy with dynamic requirements in an Agile environment. - As a Lead, conducted 1:1, Performance reviews, controlled and motivated people, and set SMART goals, helped several subordinates to grow up to seniors and leads. - Received the “QA Engineer of the Year” accolade several times. - Leading a successful company-level QA onboarding project, which reduced hiring efforts by 50%.

- Proactively minimized production defects through meticulous testing practices, resulting in a demonstrably positive impact on product quality and customer experience. - Led multiple projects as a solo QA, both manual and automation on TestComplete and SOAP UI. - Was selected as the best QA for a challenging project among 100+ QA in the company.

- For multiple projects, was fully responsible for test plans, test cases, defect management, and test coverage, using the most appropriate test design methods. - Pioneered the implementation of an Agile testing approach within the project.

- Being the only QA in the company, I pioneered the QA processes and practices - Negotiated and started the usage of testing documentation and bug tracking system - Performed a full-fledged testing of a box B2B product focusing on quality and customer needs