Optimizing Existing NLP Tasks by Leveraging Knowledge Graphs
Restricted (Penn State Only)
- Author:
- Swaminathan, Ashwath Raghav
- Graduate Program:
- Computer Science and Engineering
- Degree:
- Master of Science
- Document Type:
- Master Thesis
- Date of Defense:
- July 05, 2023
- Committee Members:
- Chitaranjan Das, Program Head/Chair
Vijaykrishnan Narayanan, Thesis Advisor/Co-Advisor
Rui Zhang, Committee Member
Abhinav Verma, Committee Member - Keywords:
- Natural Language Processing
Knowledge Graphs
Knowledge Graph Embeddings
Transformers
Computational Complexity
Memory Usage Optimization
Reducing FLOPs
Token Pruning - Abstract:
- With the introduction of transformer-based models, Natural Language Processing (NLP) has made considerable strides. By efficiently capturing contextual information and exhibiting excellent transferability across multiple language-related tasks, transformers have revolutionized NLP. In particular when processing lengthy sequences, this work examines the computational complexity and token bottleneck issues that transformer-based models confront. To address these challenges, we propose two methods: feeding custom Knowledge Graph Embeddings to a transformer-based model and feeding extracted Knowledge Graph triplets as pruned tokens. We introduce a pipeline/framework for converting sentences to knowledge graphs, where the main subject, object, and verb form the entities/nodes, and the relationships/edges comprise the knowledge graph. We also present a pipeline to convert knowledge graphs into custom Knowledge Graph Embeddings, which are then utilized by the transformer-based model. Additionally, we propose an alternative pipeline that feeds the extracted knowledge graph triplets as sparse information/pruned tokens to the transformer-based model. Through experimental studies, we evaluate the effectiveness of our proposed approaches in reducing space and computation complexity. The results highlight the potential of leveraging Knowledge Graphs to enhance the performance and scalability of transformer-based models in Natural Language Processing applications.