Increased encryption a double-edged sword

While there isn't much the DHS and the NSA can do to stem the spread of encryption, enterprises can take steps to ensure that encryption is benefiting the organization and not their enemies

security encryption key
Credit: Thinkstock

The Department of Homeland Security may be complaining that encryption is interfering with their attempts to eavesdrop on criminals and terrorists -- but they're not the only ones inconvenienced.

Enterprise security may also be hampered by ubiquitous encryption because it makes it harder to spot when sensitive data is leaving the company, or when malware is communicating with its command and control servers.

Cloud-based email services, online storage sites, and other cloud application providers have already begun encrypting traffic. Even social networks are encrypting traffic. And, recently, Firefox experimented with expanding encryption to more and more of the Web.

According to a recent report from Dell, the amount of encrypted traffic through corporate firewalls has doubled over the past year and now accounts for 60 percent of all communications.

But not all corporations are prepared for the day when all traffic in and out of their networks is encrypted.

Fortunately, while there isn't much the DHS and the NSA can do to stem the spread of encryption, there are steps that enterprises can take to ensure that encryption is benefiting them and not their enemies.

The latest smart firewalls can decrypt and monitor both incoming and outgoing traffic, helping companies with both data loss prevention and malware control.

Time for firewall upgrades

Ubiquitous encryption will most affect enterprises running older firewalls and data loss prevention solutions.

"If browsers start encrypting everything, it will make spotting malicious traffic harder," said Jeremy Scott, senior research analyst at Solutionary.

Security analysts need to be able to see traffic in order to see whether sensitive data is being exfiltrated or malware is being downloaded.

"By protecting the traffic from prying eyes, it also prevents detection of malicious activity by watchful eyes," said Scott.

Jeremy Scott, senior research analyst at Solutionary

Legacy systems are already effectively blind to 60 percent of all traffic, said John Gordineer, director of product marketing for network security at Dell.

"This is definitely a big issue," said John Pirc, chief strategy officer and co-founder at security vendor Bricata.

However, solutions are already on the market, in the form of web proxies and other security devices that provide a decryption method that allows the security appliance to inspect encrypted traffic.

In the past, systems that decrypted and analyzed traffic may have had a performance impact on communications, Pirc said.

"You need to buy dedicated devices that do SSL offload," he said.

Another option is cloud-based Web application firewalls, though these require that enterprises give up their SSL keys to an outside vendor.

Rogue encryption a big red flag

But a proxy can't always decrypt all traffic in and out of a company.

For example, employees using their own encryption programs, unsanctioned by the enterprise, can still encrypt documents before sending them out to cloud storage, to file sharing sites, to their personal email accounts, or to untrustworthy third parties.

And malware programs that use encryption to hide malicious traffic aren't likely to share keys with enterprise firewalls.

But even if this traffic can't be decrypted, the very existence of unsanctioned encrypted traffic is a warning sign, said Bryan Simon, SANS Certified Instructor at SANS Institute and president and CEO of Xploit Security.

"The proxy knows when it can't read the traffic," he said. "That's an indicator of a bad connection."

In addition, malware might not even know that it's supposed to go through the proxy, he added.

"If you have continuous networking monitoring in your environment, you can tell if your clients are launching legitimate SSL connections versus malware," he said.

Of course, malware writers will adapt, he added.

"The adversary will use what they can," he said. "If they can't do encryption, they'll go plain text. They don't care. They're that brazen."

Monitor traffic destinations -- and origins

Another approach for handling encrypted traffic is to look for suspicious destinations, said Jason Lewis, chief collection and intelligence officer at Lookingglass.

"The attacker still needs to move the data out of the enterprise and they commonly use known malicious infrastructure as the destination," he said. "If the security team is unable to look inside the traffic, they can still observe where the traffic is going."

In addition to known malicious sites, enterprises can also look for other signs that the traffic isn't legitimate.

"For example, if encrypted traffic is headed to the Tor network, it's an easy decision to disallow that communication," he said.

Companies can also look at where the traffic is coming from. Not all enterprise systems need to be communicating with the outside world, said Jean-Philippe Taggart, senior security researcher at Malwarebytes Labs.

"Sometimes it isn’t about if the traffic is encrypted, but if there should be any traffic emanating at all," he said.

The amount of data being transferred can also be a clue, said Muddu Sudhakar, co-founder and CEO at security firm Caspida.

"A large amount of data transferred to an unfamiliar IP address is potential threat indicator," he said.

Be ware of privacy concerns

When employees have an expectation of privacy when it comes to, say, their personal use of the Internet while at work, then a warning from the IT department that they're violating corporate communication policies can be an unwelcome shock.

"CSOs should communicate clearly with the staff that their traffic is actively monitored," said Malwarebytes' Taggart.

He suggested that companies have employees acknowledge that they understand that their communications are monitored when they log in to corporate systems.

"This shows the enterprise is upfront, and has the added benefit of reducing frivolous use of corporate equipment and bandwidth," he said.

Companies can also choose to not inspect traffic going to particular destinations, such as personal finance sites, said Bricata's Pirc.

"People aren't going to be comfortable for their personal banking stuff to be inspected," he said. "You might have a white list, not to decrypt for Facebook, Twitter, or anything personal."

This story, "Increased encryption a double-edged sword" was originally published by CSO.