Wildcard DNS Aggregation for Subdomain APIs

[TrebledJ]

Recently, I was scanning a domain and chaos API returned 999,999 DNS items. This is not uncommon among certain cloud or network providers. Unfortunately, these domains are processed individually which very easily chokes the event pipeline and lead to large memory consumption. Also logging takes forever to print one domain per line. I don't want my scanner to be DoSed by a big corpo. :(

I imagine (citation needed) this doesn't only apply to the chaos module, but also to other subdomain modules.

I think it would be good to apply the wildcard detection logic or just use a simple threshold, to mark such subdomains as wildcards.

---

Sample Command and Output:

└─$ bbot -t example.com -m chaos -c modules.chaos.api_key=...
  ______  _____   ____ _______
 |  ___ \|  __ \ / __ \__   __|
 | |___) | |__) | |  | | | |
 |  ___ <|  __ <| |  | | | |
 | |___) | |__) | |__| | | |
 |______/|_____/ \____/  |_|
 BIGHUGE BLS OSINT TOOL v2.8.0

www.blacklanternsecurity.com/bbot

[INFO] Scan seeded with 1 targets (1 in whitelist)
[INFO] Loaded 1/1 scan modules (chaos)
[INFO] Loaded 6/6 internal modules (aggregate,cloudcheck,dnsresolve,excavate,speculate,unarchive)
[INFO] Loaded 5/5 output modules, (csv,json,python,stdout,txt)
[INFO] internal.speculate: No portscanner enabled. Assuming open ports: 80, 443
[SUCC] chaos: API is ready
[SUCC] Setup succeeded for 14/14 modules.

[SUCC] Scan ready. Press enter to execute some_scan
[SUCC] Starting scan some_scan
[SCAN]                  some_scan (SCAN:alkdsjfljdsflkjalksjflkajdslkfjafd)      TARGET
[DNS_NAME]              example.com  TARGET  (a-record, aaaa-record, domain, in-scope, mx-record, ns-record, soa-record, target, txt-record)
[ORG_STUB]              example      speculate
[INFO] Encountered domain with wildcard DNS (TXT): *.example.com
[DNS_NAME]              aaaaaa.example.com TXT     (a-record, in-scope, subdomain)
[DNS_NAME]              bbbbbb.example.com  TXT     (a-record, in-scope, subdomain)
[DNS_NAME]              cccccc.example.com   MX      (a-record, in-scope, subdomain)
[DNS_NAME]              ns1.example.com      NS      (a-record, in-scope, subdomain)
[DNS_NAME]              ns2.example.com      NS      (a-record, in-scope, subdomain)
[DNS_NAME]              ns2.example.com      SOA     (a-record, in-scope, subdomain)
[DNS_NAME]              dddddd.example.com   MX      (a-record, in-scope, subdomain)
[DNS_NAME]              eeeeee.example.com  TXT     (in-scope, subdomain, txt-record, txt-wildcard-possible, wildcard-possible)
[DNS_NAME]              ffffff.example.com  TXT     (in-scope, subdomain, txt-record, txt-wildcard-possible, wildcard-possible)
[DNS_NAME]              gggggg.example.com        TXT     (a-record, in-scope, subdomain, txt-record, txt-wildcard-possible, wildcard-possible)
[DNS_NAME]              a.subdomain.example.com     PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              example.com  speculate       (a-record, aaaa-record, domain, in-scope, mx-record, ns-record, soa-record, txt-record)
[DNS_NAME]              hhhhhh.example.com PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              iiiiii.example.com       PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              jjjjjj.example.com      PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              kkkkkk.example.com       PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              llllll.example.com      PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              mmmmmm.example.com      PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              nnnnnn.example.com      PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              oooooo.example.com       PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              pppppp.example.com       PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              qqqqqq.example.com       PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              rrrrrr.example.com        PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              ssssss.example.com       PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              tttttt.example.com        PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              uuuuuu.example.com       PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              _wildcard.example.com        PTR     (in-scope, subdomain, txt-record, txt-wildcard, wildcard)
[DNS_NAME]              vvvvvv.example.com PTR     (a-record, in-scope, subdomain)
[DNS_NAME]              wwwwww.example.com      PTR     (a-record, in-scope, subdomain)
...
(Output goes on for hundreds of thousands of lines.)
...

If you want to reproduce/test, you can try finding or mocking an API response which returns one million domains.

[TheTechromancer]

Thanks for reporting. Were the hosts correctly identified as wildcards or no?

[TheTechromancer]

We've tuned our wildcard detection over time, but no detection is perfect and some domains' DNS is too wack to detect reliably. If you could provide some sample DNS responses (i.e. from 10-20 random hosts like asdf.evilcorp.com), it might shine some light on the issue.