Skip to main content

Secret Key Vulnerabilities

Overview

You can find results about secret key vulnerabilities in scanned projects. Typically, TruffleHog queries are used to search through repositories, including the entire commit history and branches, for strings that could represent secrets. Checkmarx Query Language (CxQL) can be used to emulate the Regex and High Shannon Entropy TruffleHog queries on scan projects.

The following queries are available in CxQL:

  • TruffleHog_HighEntropy_Strings

  • TruffleHog_Regex_Matches

  • Remove_FalseComments_TruffleHog

Caution

These queries have a tendency to generate many false positives. Before implementing them, consider weighing the value of spending time marking several false positives as "Non-Exploitable" against perhaps finding a few true positives that reveal publicly exposed secrets. Some secrets will be benign; others will have the potential for granting administrative access to your production system.

Installation

These queries are not specific to any particular language, and therefore they can be used in scans for multiple language types. The downside is that you have to install it under the queries for each of languages that you want to check.

  1. In CxAudit, create a group under Corp where the query will live. This was tested in Java, but might apply to JavaScript and other languages. For example: Corp/{Company Name}

  2. Under the created group, make a query with a name that corresponds to the name of each .cxql file included with this package. Paste the contents of each file into the corresponding query you created. (For example, Resolve_Code_And_Comment_Flows.)

  3. Set the properties on each query as follows:

    • Potential_Hardcoded_Passwords:

      • Set severity to the desired level

      • Choose categories as needed (likely fits in Sensitive Data Exposure categories)

      • CWE ID 798

      • Check the "Executable" checkbox

    • TruffleHog_Regex_Matches:

      • Set severity to the desired level

      • Choose categories as needed (likely fits in Sensitive Data Exposure categories)

      • CWE ID 798

      • Check the "Executable" checkbox

    • TruffleHog_HighEntropy_Strings:

      • Set severity to the desired level

      • Choose categories as needed (likely fits in Sensitive Data Exposure categories)

      • CWE ID 798

      • Check the "Executable" checkbox

    • Resolve_Code_And_Comment_Flows:

      • Only uncheck the "Executable" checkbox. All other properties are not applicable.

  4. Save the queries and exit CxAudit. Adjust your presets to include your new TruffleHog queries.

  5. (Optional) Execute the SQL script in file "Proc_AddFileExtToLanguage.sql" in the Checkmarx SAST database (CxDB). This adds the required stored procedure to the SQL server for adding the file extensions that are included in a scan. If there are no additional file types needed for inclusion, then this step is not necessary.

  6. (Optional) Execute the following SQL commands to add supported extensions. Modify as needed to support extensions for your language of choice.

    	exec AddFileExtToLanguage 'Java', 'properties', 'JAVA_XML_EXTENSIONS'
    	exec AddFileExtToLanguage 'Java', 'yaml', 'JAVA_XML_EXTENSIONS'
    	exec AddFileExtToLanguage 'Java', 'yml', 'JAVA_XML_EXTENSIONS'
    	exec AddFileExtToLanguage 'Java', 'json', 'JAVA_XML_EXTENSIONS'
    	
    	exec AddFileExtToLanguage 'JavaScript', 'properties', 'JS_XML_EXTENSIONS'
    	exec AddFileExtToLanguage 'JavaScript', 'yaml', 'JS_XML_EXTENSIONS'
    	exec AddFileExtToLanguage 'JavaScript', 'yml', 'JS_XML_EXTENSIONS'
    
    	exec AddFileExtToLanguage 'PHP', 'properties', 'PHP_COMMON_EXTENSIONS'
    	exec AddFileExtToLanguage 'PHP', 'yaml', 'PHP_COMMON_EXTENSIONS'
    	exec AddFileExtToLanguage 'PHP', 'yml', 'PHP_COMMON_EXTENSIONS'
    	exec AddFileExtToLanguage 'PHP', 'json', 'PHP_COMMON_EXTENSIONS'
    		
    	exec AddFileExtToLanguage 'Ruby', 'yaml', 'RUBY_OTHERS_EXTENSIONS'
    	exec AddFileExtToLanguage 'Ruby', 'yml', 'RUBY_OTHERS_EXTENSIONS'
    	exec AddFileExtToLanguage 'Ruby', 'json', 'RUBY_OTHERS_EXTENSIONS'
    		
    	exec AddFileExtToLanguage 'Groovy', 'yaml', 'GROOVY_EXTENSIONS'
    	exec AddFileExtToLanguage 'Groovy', 'yml', 'GROOVY_EXTENSIONS'
    	exec AddFileExtToLanguage 'Groovy', 'json', 'GROOVY_EXTENSIONS'
    	
    	exec AddFileExtToLanguage 'Scala', 'yaml', 'SCALA_EXTENSIONS'
    	exec AddFileExtToLanguage 'Scala', 'yml', 'SCALA_EXTENSIONS'
    	exec AddFileExtToLanguage 'Scala', 'json', 'SCALA_EXTENSIONS'
    	
    	exec AddFileExtToLanguage 'Python', 'yaml', 'PYTHON_CORE_EXTENSIONS'
    	exec AddFileExtToLanguage 'Python', 'yml', 'PYTHON_CORE_EXTENSIONS'
    	exec AddFileExtToLanguage 'Python', 'json', 'PYTHON_CORE_EXTENSIONS'
    

    These commands will add ".properties", ".yml", ".yaml", and ".json" files to the scan so the queries can execute Regex searches on the contents.

  7. Restart all IIS servers.

  8. Adjust scanning presets on projects to include the new rules added in CxAudit.

  9. Perform scans and inspect vulnerabilities reported with the new queries.

General Comments

Potential_Hardcoded_Passwords

Searching for password fields in files without the specific format of the sensitive strings will be prone to many false-positive and false-negatives.

The reason for false-positives is that there might be variable names matching the search criteria that are not actually related to passwords or other sensitive data.

It is also possible that the config key values and variable names were changed to formats that are not recognized as elements that typically expose sensitive data.

Query Details

TruffleHog_HighEntropy_Strings

This query uses entropy in order to detect secrets. The entropy of a random variable is the average level of information, surprise, or uncertainty inherent in the variable's possible outcomes. Shannon Entropy is used to calculate this level.

Func<string, string, double> shannon_entropy = (word, charSet) =>{
        double entropy = 0.0;

        foreach (char x in charSet)
        {
                try{
                        int countOccurences = word.Where(c => c.Equals(x)).Count();
                        double p_x = (double) countOccurences / word.Length;
                
                        if (p_x > 0)
                                entropy += -p_x * Math.Log(p_x, 2.0);
                }
                catch(Exception e){};
        }
                
        return entropy;
};

The general query FindByRegexExt is used to look for strings in all files in the project that is being analyzed. It returns Comment nodes and the strings associated with them are analyzed using the Shannon Entropy algorithm. Only strings with a minimum length of 20 characters are considered possible secrets.

int MIN_STRING_LEN = 20;

Since it is difficult to know which unknown secrets it cannot detect, this query has a tendency to generate many false positives.

In order to lower the number of false positives, two thresholds are used to filter some of those results.

double BASE64_ENTROPY = 4.5;
double HEX_ENTROPY = 2.7;

This means that the query only returns the strings in which the entropy calculated by the algorithm is higher than the specified threshold.

Notice

Although for now this query is only used in Java, it was created as a Common query because it is planned to be used by other languages.

TruffleHog_Regex_Matches

This query uses regular expressions in order to detect secrets.

A dictionary was created containing several regular expressions. They are designed to recognize the patterns of sensitive data.

Dictionary<string, string> regexes = new Dictionary<string, string>(); 
regexes.Add("Slack Token", "(xox[p|b|o|a]-[0-9]{12}-[0-9]{12}-[0-9]{12}-[a-z0-9]{32})");
...
regexes.Add("Twitter OAuth", "[t|T][w|W][i|I][t|T][t|T][e|E][r|R][^\\s]*['|\"][0-9a-zA-Z]{35,44}['|\"]");

This query returns all the strings associated with the Comment nodes found by FindByRegexExt (which is also used in this query to look for strings in all files in the project that is being analyzed) that match any of the Regular Expressions present on the dictionary.

foreach(String regexKey in regexes.Keys)
{       
        CxList found = All.FindByRegexExt(regexes[regexKey], "*.*");  
        result.Add(Remove_FalseComments_TruffleHog(found));
}

The only caveat to this is that the Regex specifying the format needs to exist and be accurate, but the results from this query will likely be more accurate because it is searching for concrete patterns that match sensitive data.

Notice

Although for now this query is only used in Java, it was created as a Common query because it is planned to be used by other languages.

Remove_FalseComments_TruffleHog

Since the general query FindByRegexExt only returns Comment Nodes and does not evaluate the DOM, but instead evaluates the code as a set of strings, a Common query was created to filter the strings

that effectively represent the real strings that will be used by the above queries.

The following nodes are removed:

  • IntegerLiterals;

  • RealLiterals;

  • MethodDecls;

  • ConstructorDecl;

  • ClassDecl;

Since it is impossible to directly compare the nodes on this list with the comments returned by FindByRegexExt, the filenames and LinePragmas are used for comparison.

When a node’s filename and a LinePragma are the same as a comment’s filename and LinePragma, it is assumed that they are the same and can be removed. Thus it is possible to filter the comments, leaving only potential secrets.