Best algorithm for parsing logs, string searching for multiple needles in haystack
Nicolas Lattuada
124

Hello everybody!
Inspired from a real world problem, I would like to start a discussion.
Let's say I want to build a library for removing sensible information from application logs.
The values are logged by pairs like the following:

[abc/servicename] [2021-02-17T11:07:35,600Z] [WARN] [ShardRecordProcessor-0283] [converters.MyClassService] {dd.env=live, dd.service=servicename, dd.version=, shard=shardId-000000000009, stream=streamname123} process=convert_to_transaction, result=transactionNotValid, transactionId='value {
  value: "XXXX-XXXXX-XXXXX-XXXXX"
}
', userId='YYYYY-YYYY-YYYYY-YYYY', password='secretvalue123', someArbitraryValue=hello123, anthing=anyvalue

I have a list of sensible terms for example, given the input above it could be: "password", "transactionId" and "anything"

I want to obfuscate the values for which the keys are contained in my sensible terms list.

So far I reckon there are at least 3 approaches:

  • Aho corrasick (link)
  • Ukkonen (link)
  • Split the haystack by delimiting "," and search in the substrings

How would you approach this problem, and why?

Comments (0)