Search For Tokens In Python List Of Tokens
def search_subtokens(subtokens, tokens):
matching_tokens = []
for token in tokens:
if any(token in subtoken for subtoken in subtokens):
matching_tokens.append(token)
return matching_tokens
# Example usage
list_corpus = ["apple", "banana", "grapefruit", "kiwi", "pear", "pineapple"]
list_search_token = ['kiwi', 'pear']
results = search_subtokens(list_search_token, list_corpus)
print(results)
In this code, the search_subtokens
function takes list_search_token
and list_corpus
as input. It iterates through each token
in the list_corpus
and checks if any of the items in list_search_token
are present in the token
using a generator expression inside the any()
function. If any of the items are found, the token
is added to the matching_tokens
list.